Oct 30 17:57:39.380573 ip-10-0-128-246 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:57:39.768563 ip-10-0-128-246 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:39.768563 ip-10-0-128-246 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:57:39.768563 ip-10-0-128-246 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:39.768563 ip-10-0-128-246 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:39.771557 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771468 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:39.771557 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771557 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771561 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771565 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771568 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771570 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771573 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771576 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771578 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771582 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771584 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771587 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771589 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771592 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771594 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771597 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771599 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771602 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771604 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771607 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771609 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:39.771622 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771612 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771620 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771623 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771626 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771629 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771634 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771638 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771641 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771643 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771646 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771648 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771651 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771653 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771655 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771658 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771660 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771663 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771665 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771668 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:39.772087 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771670 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771673 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771675 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771678 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771680 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771683 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771685 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771687 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771690 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771692 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771695 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771697 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771700 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771703 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771705 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771709 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771711 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771714 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771717 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:39.772559 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771719 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771722 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771725 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771727 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771730 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771732 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771735 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771737 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771740 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771742 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771745 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771747 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771749 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771752 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771754 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771756 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771759 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771761 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771764 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:39.772999 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771766 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771768 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771771 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771774 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771776 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771778 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771782 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771785 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771787 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771790 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771793 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771796 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771798 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771801 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771803 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771806 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771809 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771813 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771815 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:39.773457 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771819 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771822 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771824 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771827 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771829 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771832 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771834 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771837 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.771839 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772658 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772665 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772667 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772670 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772673 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772676 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772679 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772681 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772684 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772686 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772689 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:39.773895 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772692 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772695 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772699 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772702 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772710 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772713 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772715 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772718 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772721 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772725 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772729 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772733 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772736 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772738 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772741 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772744 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772746 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772749 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772751 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772754 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:39.774377 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772756 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772759 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772761 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772764 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772766 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772769 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772771 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772774 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772777 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772779 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772781 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772784 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772786 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772789 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772791 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772794 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772796 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772805 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772807 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:39.774859 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772810 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772812 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772815 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772817 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772819 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772822 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772824 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772826 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772829 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772832 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772834 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772837 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772839 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772842 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772845 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772847 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772849 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772853 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772856 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:39.775313 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772859 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772861 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772863 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772866 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772868 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772871 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772874 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772876 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772879 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772885 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772888 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772890 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772898 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772901 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772903 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772906 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772908 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772911 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772914 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772916 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:39.775774 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772919 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772922 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772925 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772927 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772937 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772940 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772942 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772945 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772947 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772950 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772953 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772955 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772957 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772961 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772963 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772966 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.772968 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773556 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773562 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:39.776394 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773565 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773567 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773570 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773573 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773575 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773583 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773587 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773590 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773592 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773595 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773597 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773600 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773602 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773605 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773608 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773611 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773613 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773616 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773618 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773620 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:39.777115 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773623 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773625 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773628 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773630 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773633 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773635 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773638 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773640 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773643 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773645 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773648 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773651 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773653 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773656 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773658 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773661 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773663 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773665 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773677 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:39.777956 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773680 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773682 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773685 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773688 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773690 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773693 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773695 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773698 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773700 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773703 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773705 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773708 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773710 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773713 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773715 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773718 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773720 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773722 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773725 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773727 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:39.778787 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773731 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773733 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773736 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773738 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773740 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773743 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773745 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773748 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773750 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773753 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773755 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773758 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773766 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773769 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773772 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773774 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773776 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773779 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773781 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:39.779349 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773784 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773786 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773789 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773791 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773794 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773796 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773799 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773801 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773803 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773806 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773808 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773811 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773813 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773816 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773818 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773822 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773825 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773828 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773830 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:39.780028 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773832 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773835 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773837 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773840 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773844 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773848 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.773853 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775543 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775553 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775562 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775566 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775571 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775574 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775578 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775582 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775585 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775588 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775592 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775596 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775599 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775601 2576 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775604 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:57:39.780535 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775607 2576 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775610 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775612 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775619 2576 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775622 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775625 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775628 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775631 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775635 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775638 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775647 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775650 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775656 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775659 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775662 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775665 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775669 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775673 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775684 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775687 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775690 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775693 2576 flags.go:64] FLAG: --enable-server="true" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775696 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775702 2576 flags.go:64] FLAG: --event-burst="100" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775705 2576 flags.go:64] FLAG: --event-qps="50" Oct 30 17:57:39.781236 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775708 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775711 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775714 2576 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775718 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775721 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775724 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775727 2576 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775730 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775733 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775735 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775738 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775741 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775744 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775746 2576 flags.go:64] FLAG: --feature-gates="" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775750 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775754 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775757 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775760 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775763 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775767 2576 flags.go:64] FLAG: --help="false" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775770 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-246.ec2.internal" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775773 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775776 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:57:39.781949 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775780 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775783 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775787 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775797 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775800 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775803 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775806 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775809 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775812 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775815 2576 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775818 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775821 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775824 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775827 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775830 2576 flags.go:64] FLAG: --lock-file="" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775832 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775835 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775838 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775844 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775847 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775849 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775852 2576 flags.go:64] FLAG: --logging-format="text" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775855 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775858 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:57:39.782526 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775861 2576 flags.go:64] FLAG: --manifest-url="" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775864 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775869 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775872 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775876 2576 flags.go:64] FLAG: --max-pods="110" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775879 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775882 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775885 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775888 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775891 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775894 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775897 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775909 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775912 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775915 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775918 2576 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775921 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775923 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775926 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775929 2576 flags.go:64] FLAG: --port="10250" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775932 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775934 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05548694925729995" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775937 2576 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775940 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775943 2576 flags.go:64] FLAG: --register-node="true" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775946 2576 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:57:39.783093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775950 2576 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775952 2576 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775955 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775957 2576 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775961 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775964 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775967 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775969 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775972 2576 flags.go:64] FLAG: --runonce="false" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775975 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775978 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775981 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775984 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775986 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775989 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775992 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775995 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.775998 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776001 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776044 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776303 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776316 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776326 2576 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776335 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776352 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:57:39.783735 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776360 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776368 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776430 2576 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776438 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776447 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776456 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776464 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776473 2576 flags.go:64] FLAG: --v="2" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776534 2576 flags.go:64] FLAG: --version="false" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776545 2576 flags.go:64] FLAG: --vmodule="" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776561 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.776569 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777501 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777517 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777524 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777529 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777535 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777540 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777545 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777549 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777554 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777558 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:39.784409 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777562 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777566 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777570 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777575 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777579 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777583 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777587 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777591 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777596 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777600 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777604 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777608 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777612 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777616 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777620 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777625 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777629 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777635 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777639 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:39.784942 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777643 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777648 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777652 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777656 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777661 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777666 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777670 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777675 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777679 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777683 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777687 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777694 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777701 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777707 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777711 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777716 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777721 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777725 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777729 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:39.785439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777734 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777738 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777742 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777746 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777751 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777756 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777760 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777764 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777769 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777773 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777777 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777781 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777785 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777789 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777794 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777798 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777803 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777819 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777826 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777832 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:39.785893 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777838 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777842 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777847 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777851 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777855 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777859 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777864 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777868 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777872 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777876 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777880 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777885 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777889 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777893 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777897 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777901 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777905 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777909 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777913 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777917 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:39.786383 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777922 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777927 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777931 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777935 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777939 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777943 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777947 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777951 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777955 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777961 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777967 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777971 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777975 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777979 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777983 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777988 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777992 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:39.786853 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.777998 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.778042 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778104 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778113 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778118 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778122 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778126 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778131 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778135 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778140 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778144 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778148 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778152 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778156 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778162 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:39.787243 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778166 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778170 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778174 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778178 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778182 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778187 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778191 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778195 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778199 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778209 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778213 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778219 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778223 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778227 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778231 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778235 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778240 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778244 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778248 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:39.787636 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778252 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778256 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778282 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778287 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778291 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778295 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778302 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778309 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778314 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778319 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778323 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778327 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778331 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778336 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778341 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778345 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778349 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778353 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778358 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:39.788078 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778361 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778366 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778371 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778381 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778386 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778390 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778395 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778399 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778403 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778407 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778411 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778416 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778420 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778428 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778435 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778439 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778443 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778448 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778452 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778456 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:39.788601 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778460 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778464 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778468 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778472 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778476 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778480 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778484 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778488 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778493 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778497 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778501 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778506 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778510 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778513 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778518 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778525 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778530 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778534 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778538 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778543 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:39.789085 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778547 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778551 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778555 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778559 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778563 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778567 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778572 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778576 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778580 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778584 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778588 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778592 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778597 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778603 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.778607 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.785663 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.785678 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.785698 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.785703 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785762 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:39.789562 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785767 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785770 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785773 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785776 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785778 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785781 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785784 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785786 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785789 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785792 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785794 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785797 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785799 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785802 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785804 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785807 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785809 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785812 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785814 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785817 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:39.790038 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785819 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785822 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785824 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785827 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785829 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785831 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785834 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785837 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785841 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785843 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785846 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785848 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785851 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785853 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785856 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785858 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785861 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785864 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785866 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:39.790520 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785869 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785871 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785874 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785876 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785879 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785883 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785887 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785891 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785894 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785897 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785901 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785903 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785906 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785908 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785911 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785913 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785916 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785919 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785921 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:39.791025 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785924 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785926 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785929 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785932 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785935 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785938 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785940 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785943 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785945 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785947 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785950 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785952 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785955 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785957 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785960 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785962 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785965 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785968 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785972 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:39.791502 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785974 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785977 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785979 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785982 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785984 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785986 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785989 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785991 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785994 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785996 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.785999 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786001 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786004 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786006 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786009 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786012 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786014 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786017 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786020 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786022 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:39.791950 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786025 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786027 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786030 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786032 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786035 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786037 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786040 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786042 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.786069 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786204 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786208 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786211 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786213 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786216 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786218 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786221 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:39.792429 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786224 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786226 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786230 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786234 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786238 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786241 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786243 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786246 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786248 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786251 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786253 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786255 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786275 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786279 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786283 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786285 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786288 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786290 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:39.792824 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786293 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786296 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786298 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786301 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786304 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786306 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786309 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786311 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786314 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786316 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786319 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786321 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786324 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786326 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786329 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786331 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786334 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786336 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786339 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786341 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:39.793244 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786343 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786346 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786348 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786351 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786353 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786357 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786360 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786363 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786366 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786369 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786372 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786374 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786377 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786379 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786382 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786384 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786387 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786390 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786392 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:39.793730 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786395 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786397 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786400 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786402 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786405 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786407 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786409 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786412 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786414 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786417 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786420 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786422 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786425 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786427 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786430 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786432 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786434 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786437 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786439 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786442 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:39.794222 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786444 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786447 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786449 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786452 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786455 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786457 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786459 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786462 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786464 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786467 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786469 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786472 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786474 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786476 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786479 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786481 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786483 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786486 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786488 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:39.794720 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786491 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786493 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786496 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.786519 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786551 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786555 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786558 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786561 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786563 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786566 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786568 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786571 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786573 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786575 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786578 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786580 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:39.795179 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786583 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786585 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786588 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786592 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786595 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786597 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786600 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786602 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786605 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786609 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786611 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786614 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786616 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786619 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786621 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786624 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786626 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786631 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786633 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:39.795587 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786636 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786639 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786641 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786644 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786646 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786649 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786651 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786653 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786656 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786658 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786661 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786664 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786666 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786669 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786671 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786674 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786678 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786680 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786683 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786685 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:39.796029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786688 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786690 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786693 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786695 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786698 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786700 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786703 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786705 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786707 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786710 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786714 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786717 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786719 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786722 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786725 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786727 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786730 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786732 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786735 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:39.796637 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786737 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786740 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786742 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786745 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786747 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786750 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786752 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786755 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786757 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786762 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786765 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786767 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786770 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786772 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786775 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786779 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786782 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786785 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786788 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:39.797089 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786790 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786793 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786795 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786798 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786801 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786804 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786806 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786809 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786812 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786814 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786817 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786819 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786822 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786824 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786827 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786829 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:39.786832 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.786886 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.788956 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:57:39.797585 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.789843 2576 server.go:1014] "Starting client certificate rotation" Oct 30 17:57:39.798074 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.789941 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:57:39.798074 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.790728 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:57:39.798074 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.791767 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:57:39.798074 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.792829 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:57:39.803226 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.803213 2576 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:57:39.808345 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.808329 2576 log.go:25] "Validated CRI v1 image API" Oct 30 17:57:39.809119 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.809093 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:57:39.813302 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.813283 2576 fs.go:135] Filesystem UUIDs: map[4f255b88-c943-4a4e-964a-1b5d252f5ff3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e80626bd-5894-4990-beb4-8b6fb3ae231f:/dev/nvme0n1p3] Oct 30 17:57:39.813362 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.813302 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:57:39.818381 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.818279 2576 manager.go:217] Machine: {Timestamp:2026-10-30 17:57:39.817792657 +0000 UTC m=+0.322435787 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3150311 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27f2bdfb8cb239bbb22f16f0d990a0 SystemUUID:ec27f2bd-fb8c-b239-bbb2-2f16f0d990a0 BootID:c3fce4a0-5e77-409a-9298-b43e8785ded3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:24:9a:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:24:9a:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:69:a6:2b:fc:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:57:39.818805 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.818795 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:57:39.818906 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.818894 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:57:39.821352 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.821326 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:57:39.821476 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.821354 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-246.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:57:39.821522 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.821478 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:57:39.821522 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.821486 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:57:39.821522 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.821511 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:57:39.822977 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.822967 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:57:39.823746 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.823736 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:57:39.827421 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.827409 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:57:39.827547 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.827538 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:57:39.828291 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.828282 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:57:39.828322 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.828296 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:57:39.828322 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.828307 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:57:39.828322 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.828315 2576 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:57:39.828322 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.828323 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:57:39.828616 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.828605 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:57:39.830680 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.830665 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:57:39.835174 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.835162 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:57:39.835936 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.835925 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:57:39.835974 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.835955 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:57:39.836108 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836098 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:57:39.836137 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836114 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:57:39.836137 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836120 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:57:39.836137 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836125 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:57:39.836137 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836130 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:57:39.836137 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836136 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:57:39.836706 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836695 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:57:39.836706 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836707 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:57:39.836770 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836715 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:57:39.836770 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836721 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:57:39.836770 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836730 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:57:39.836770 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836738 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:57:39.836770 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836765 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:57:39.836770 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.836771 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:57:39.840233 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.840222 2576 server.go:1263] "Started kubelet" Oct 30 17:57:39.840374 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.840340 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:57:39.840374 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.840342 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:57:39.840494 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.840403 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:57:39.840964 ip-10-0-128-246 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:57:39.841562 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.841541 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:57:39.844055 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.844041 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:57:39.846927 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.846896 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:57:39.848655 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.848637 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:57:39.849168 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.849148 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:57:39.849284 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.849250 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:39.849713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.849673 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:57:39.849713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.849676 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:57:39.849713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.849701 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:57:39.849885 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.849833 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:57:39.849885 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.849844 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:57:39.850018 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.850005 2576 factory.go:55] Registering systemd factory Oct 30 17:57:39.850061 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.850022 2576 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:57:39.850292 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.850274 2576 factory.go:153] Registering CRI-O factory Oct 30 17:57:39.850292 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.850293 2576 factory.go:223] Registration of the crio container factory successfully Oct 30 17:57:39.850414 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.850346 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:57:39.850414 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.850364 2576 factory.go:103] Registering Raw factory Oct 30 17:57:39.850414 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.850376 2576 manager.go:1199] Started watching for new ooms in manager Oct 30 17:57:39.851296 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.851280 2576 manager.go:319] Starting recovery of all containers Oct 30 17:57:39.862043 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.862025 2576 manager.go:324] Recovery completed Oct 30 17:57:39.863831 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.863809 2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 17:57:39.867203 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.867189 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:39.868886 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.868863 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:39.868958 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.868898 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:39.868958 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.868907 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:39.869545 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.869528 2576 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:57:39.869545 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.869539 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:39.869681 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.869540 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:57:39.869681 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.869602 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:57:39.870156 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.870143 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:39.870195 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.870164 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:39.870195 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.870176 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:39.871712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.871700 2576 policy_none.go:50] "Start" Oct 30 17:57:39.871762 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.871716 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:57:39.871762 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.871729 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:57:39.873708 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.873698 2576 policy_none.go:44] "Start" Oct 30 17:57:39.878854 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.878825 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:57:39.878948 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.878830 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-246.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:57:39.879921 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.878835 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-246.ec2.internal.18e3606423fa64d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-246.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-246.ec2.internal,},FirstTimestamp:2026-10-30 17:57:39.840197846 +0000 UTC m=+0.344840975,LastTimestamp:2026-10-30 17:57:39.840197846 +0000 UTC m=+0.344840975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-246.ec2.internal,}" Oct 30 17:57:39.887932 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.887828 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:57:39.888005 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.887881 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lm9df" Oct 30 17:57:39.892143 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.892115 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:57:39.892143 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.892132 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:57:39.915802 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.915771 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:57:39.917997 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.917976 2576 manager.go:341] "Starting Device Plugin manager" Oct 30 17:57:39.918074 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.918012 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:57:39.918074 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.918024 2576 server.go:87] "Starting device plugin registration server" Oct 30 17:57:39.918292 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.918278 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:57:39.918568 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.918293 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:57:39.918568 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.918370 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:57:39.918568 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.918433 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:57:39.918568 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.918441 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:57:39.919052 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.919030 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:57:39.919132 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.919084 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:39.954643 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.954621 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:39.974123 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.974094 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:57:39.975355 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.975340 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:57:39.975434 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.975362 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:57:39.975434 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.975382 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:57:39.975434 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:39.975417 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 17:57:39.991658 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:39.991637 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:40.019199 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.019151 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:40.019939 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.019925 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:40.020006 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.019951 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:40.020006 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.019961 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:40.020006 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.019984 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.037681 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.037664 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.037726 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.037684 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-246.ec2.internal\": node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.076479 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.076059 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"] Oct 30 17:57:40.076552 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.076514 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:40.077012 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.076985 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.077277 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.077243 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:40.077367 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.077291 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:40.077367 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.077308 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:40.079555 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.079542 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:40.079725 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.079712 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.079780 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.079738 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:40.081953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.081936 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:40.081953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.081944 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:40.082107 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.081965 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:40.082107 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.081975 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:40.082107 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.081965 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:40.082107 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.082056 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:40.084196 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.084175 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.084282 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.084202 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:40.084895 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.084881 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:40.084970 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.084911 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:40.084970 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.084925 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:40.101851 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.101835 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-246.ec2.internal\" not found" node="ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.106914 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.106899 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-246.ec2.internal\" not found" node="ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.151961 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.151940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46fe009f9a7f834e879e5480aab152d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" (UID: \"46fe009f9a7f834e879e5480aab152d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.152098 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.151966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.152098 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.151983 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.177071 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.177043 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.252073 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.252053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46fe009f9a7f834e879e5480aab152d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" (UID: \"46fe009f9a7f834e879e5480aab152d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.252201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.252085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.252201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.252107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.252201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.252159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.252201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.252161 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.252201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.252178 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46fe009f9a7f834e879e5480aab152d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" (UID: \"46fe009f9a7f834e879e5480aab152d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.277357 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.277310 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.377399 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.377378 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.404917 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.404899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.410646 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.410620 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:57:40.477581 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.477549 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.578050 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.578003 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.678473 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.678452 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.778982 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.778957 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.790128 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.790109 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:57:40.790253 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.790235 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:57:40.790313 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.790272 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:57:40.879889 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:40.879869 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:57:40.892634 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.892600 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:52:40 +0000 UTC" deadline="2028-05-22 14:37:05.230025456 +0000 UTC" Oct 30 17:57:40.892634 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.892628 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13676h39m24.337400892s" Oct 30 17:57:40.951277 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:40.951233 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:41.005620 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:41.004656 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fe009f9a7f834e879e5480aab152d6.slice/crio-7f4e39ad52d0ec41bac2e39cda30df3b4208773218386c5ffd2f44f991528f5a WatchSource:0}: Error finding container 7f4e39ad52d0ec41bac2e39cda30df3b4208773218386c5ffd2f44f991528f5a: Status 404 returned error can't find the container with id 7f4e39ad52d0ec41bac2e39cda30df3b4208773218386c5ffd2f44f991528f5a Oct 30 17:57:41.006224 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:41.006197 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67318722ef4c3e2a1b8bc7b38d57c5ad.slice/crio-0311e96784e88a5761844fee94b5e902487d37e562f249063e7a567a5c866402 WatchSource:0}: Error finding container 0311e96784e88a5761844fee94b5e902487d37e562f249063e7a567a5c866402: Status 404 returned error can't find the container with id 0311e96784e88a5761844fee94b5e902487d37e562f249063e7a567a5c866402 Oct 30 17:57:41.011706 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.011688 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:57:41.050298 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.050278 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:57:41.071877 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.071860 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:41.071954 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.071943 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:57:41.096147 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.096131 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:41.433702 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.433618 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:41.773945 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.773878 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:41.830327 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.830301 2576 apiserver.go:52] "Watching apiserver" Oct 30 17:57:41.849997 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.849969 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:57:41.856902 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.856876 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:57:41.857300 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.857278 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-q74cr","openshift-ovn-kubernetes/ovnkube-node-dx2gw","kube-system/global-pull-secret-syncer-dt49r","kube-system/konnectivity-agent-vz7gh","openshift-cluster-node-tuning-operator/tuned-zb8gk","openshift-dns/node-resolver-rv2ks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx","openshift-multus/network-metrics-daemon-hk9ss","openshift-image-registry/node-ca-6wlg8","openshift-multus/multus-ch8b8","openshift-network-operator/iptables-alerter-j8t4c","openshift-multus/multus-additional-cni-plugins-p4pv4"] Oct 30 17:57:41.860114 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.860089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:41.862363 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.862336 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:41.864747 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.864725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:41.864858 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:41.864832 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:41.868224 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.866969 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:41.868224 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:41.867037 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:57:41.870349 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.869741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.872318 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.871893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:41.874263 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.874243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:41.876663 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.876643 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:57:41.876767 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.876658 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:57:41.876835 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.876781 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p48xz\"" Oct 30 17:57:41.876900 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.876858 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:57:41.877008 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.876955 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:41.877081 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:41.877048 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:41.877177 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.877161 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:57:41.877216 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.877207 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:57:41.878917 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.878855 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzq26\"" Oct 30 17:57:41.879726 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.879706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.882184 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.882163 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.884821 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.884802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:41.887977 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.887958 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:41.918585 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.918559 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:57:41.918684 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.918625 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d4tkf" Oct 30 17:57:41.928231 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.928206 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d4tkf" Oct 30 17:57:41.951829 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.951771 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:57:41.962286 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-multus\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.962384 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-daemon-config\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.962384 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-tmp\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.962384 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-netd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.962384 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-tmp-dir\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-socket-dir-parent\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-k8s-cni-cncf-io\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-iptables-alerter-script\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f34929e-0b21-4743-af11-1360d39f162b-serviceca\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-sys\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cnibin\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zm6w\" (UniqueName: \"kubernetes.io/projected/1bc68151-0620-48d2-a803-e7cb588c5b76-kube-api-access-4zm6w\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-system-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.962578 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-cni-binary-copy\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962593 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7fzn\" (UniqueName: \"kubernetes.io/projected/4f34929e-0b21-4743-af11-1360d39f162b-kube-api-access-p7fzn\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-kubelet-config\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962713 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-bin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-kubelet\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962742 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-var-lib-kubelet\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-etc-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-node-log\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-env-overrides\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962799 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.962937 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-multus-certs\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-etc-kubernetes\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962934 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-systemd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxpz\" (UniqueName: \"kubernetes.io/projected/d79402ab-1f1f-458b-b7c8-3c8209e69efb-kube-api-access-tlxpz\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.962991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-hosts-file\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963017 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f619d36-116d-4434-bf1e-1aecb4d7fefe-konnectivity-ca\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/66e91a23-8415-48a9-924c-64f752b59272-kube-api-access-59rfw\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-slash\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-script-lib\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-socket-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-sys-fs\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qcx\" (UniqueName: \"kubernetes.io/projected/ab36263f-c154-4f9f-9a8f-10fc38405e52-kube-api-access-n7qcx\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f619d36-116d-4434-bf1e-1aecb4d7fefe-agent-certs\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:41.963597 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwv6\" (UniqueName: \"kubernetes.io/projected/c598714d-0188-42bd-80e6-d9873dc2ae20-kube-api-access-6dwv6\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-hostroot\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-host-slash\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-modprobe-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-conf\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-host\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-netns\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963445 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-bin\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlqs4\" (UniqueName: \"kubernetes.io/projected/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-kube-api-access-xlqs4\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-systemd\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-ovn\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-config\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-device-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-cnibin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-log-socket\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:41.964225 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysconfig\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-var-lib-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvqb\" (UniqueName: \"kubernetes.io/projected/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-kube-api-access-dgvqb\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7gv\" (UniqueName: \"kubernetes.io/projected/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-kube-api-access-cg7gv\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovn-node-metrics-cert\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-dbus\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-conf-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f34929e-0b21-4743-af11-1360d39f162b-host\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-kubernetes\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-systemd-units\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-system-cni-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-run\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-etc-tuned\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963945 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-kubelet\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.963963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-registration-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.964005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-netns\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:41.964927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.964039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-lib-modules\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:41.965525 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.964067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:41.967595 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.967567 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:41.981162 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.981112 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" event={"ID":"46fe009f9a7f834e879e5480aab152d6","Type":"ContainerStarted","Data":"7f4e39ad52d0ec41bac2e39cda30df3b4208773218386c5ffd2f44f991528f5a"} Oct 30 17:57:41.982671 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.982645 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" event={"ID":"67318722ef4c3e2a1b8bc7b38d57c5ad","Type":"ContainerStarted","Data":"0311e96784e88a5761844fee94b5e902487d37e562f249063e7a567a5c866402"} Oct 30 17:57:41.984199 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984118 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:57:41.984199 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984133 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:57:41.984199 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984175 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:57:41.984199 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984204 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984725 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984770 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984867 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984911 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984937 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984957 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.984997 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985010 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985032 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985021 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985081 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985090 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985131 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985139 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985153 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:57:41.985191 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985172 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:57:41.986132 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985583 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nwsqb\"" Oct 30 17:57:41.986132 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985748 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:57:41.986132 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985780 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:57:41.986132 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.985918 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt2qr\"" Oct 30 17:57:41.987007 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.986985 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9m88r\"" Oct 30 17:57:41.987100 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.987072 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wj5nd\"" Oct 30 17:57:41.987278 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.987231 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jcq9r\"" Oct 30 17:57:41.987541 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.987516 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6627\"" Oct 30 17:57:41.987671 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:41.987643 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh48h\"" Oct 30 17:57:42.064581 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwv6\" (UniqueName: \"kubernetes.io/projected/c598714d-0188-42bd-80e6-d9873dc2ae20-kube-api-access-6dwv6\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.064758 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-hostroot\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.064758 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-host-slash\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:42.064758 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064709 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-hostroot\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.064758 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-modprobe-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.064953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-conf\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.064953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064777 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-host-slash\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:42.064953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-host\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.064953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-netns\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.064953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064843 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-bin\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.064953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlqs4\" (UniqueName: \"kubernetes.io/projected/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-kube-api-access-xlqs4\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:42.064953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.064953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064903 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-modprobe-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.064953 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-systemd\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064954 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-netns\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064963 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-bin\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.064966 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-systemd\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065025 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-host\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065104 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-conf\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065128 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-ovn\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-config\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-device-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-ovn\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-cnibin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-log-socket\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:42.065265 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysconfig\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-var-lib-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvqb\" (UniqueName: \"kubernetes.io/projected/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-kube-api-access-dgvqb\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7gv\" (UniqueName: \"kubernetes.io/projected/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-kube-api-access-cg7gv\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovn-node-metrics-cert\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-dbus\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-conf-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f34929e-0b21-4743-af11-1360d39f162b-host\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-kubernetes\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065500 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-systemd-units\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-system-cni-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-run\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-etc-tuned\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-kubelet\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-registration-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-netns\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-lib-modules\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.065913 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-multus\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065785 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-daemon-config\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065789 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-config\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-tmp\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-netd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065843 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-tmp-dir\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065860 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-kubernetes\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-socket-dir-parent\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065898 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-device-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-k8s-cni-cncf-io\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-iptables-alerter-script\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065980 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-k8s-cni-cncf-io\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.065977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f34929e-0b21-4743-af11-1360d39f162b-serviceca\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066039 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-systemd-units\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-sys\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.066627 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cnibin\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zm6w\" (UniqueName: \"kubernetes.io/projected/1bc68151-0620-48d2-a803-e7cb588c5b76-kube-api-access-4zm6w\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-system-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-cni-binary-copy\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7fzn\" (UniqueName: \"kubernetes.io/projected/4f34929e-0b21-4743-af11-1360d39f162b-kube-api-access-p7fzn\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066323 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f34929e-0b21-4743-af11-1360d39f162b-serviceca\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066400 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-kubelet-config\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066438 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-system-cni-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-conf-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066496 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-socket-dir-parent\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066518 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f34929e-0b21-4743-af11-1360d39f162b-host\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066603 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cnibin\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066617 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066624 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-cnibin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066661 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-dbus\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:42.067465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066695 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-system-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066740 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-tmp-dir\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066751 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-log-socket\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-run\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.066840 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-sys\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066855 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.066900 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:42.566875858 +0000 UTC m=+3.071518980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysconfig\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066974 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-var-lib-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067245 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-cni-binary-copy\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.066319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-kubelet-config\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067538 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-kubelet\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-bin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067577 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-bin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-kubelet\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-var-lib-kubelet\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.068239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-etc-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067632 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-node-log\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-env-overrides\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-multus-certs\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-etc-kubernetes\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-systemd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-netd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxpz\" (UniqueName: \"kubernetes.io/projected/d79402ab-1f1f-458b-b7c8-3c8209e69efb-kube-api-access-tlxpz\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-hosts-file\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067847 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-multus\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.068585 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-daemon-config\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.068623 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-registration-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.069170 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.068701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-netns\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.068760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.068832 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-lib-modules\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.068898 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-kubelet\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.068935 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.068970 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-var-lib-kubelet\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069018 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-etc-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069079 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-node-log\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067631 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069118 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f619d36-116d-4434-bf1e-1aecb4d7fefe-konnectivity-ca\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/66e91a23-8415-48a9-924c-64f752b59272-kube-api-access-59rfw\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069178 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-env-overrides\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-slash\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069194 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-script-lib\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069251 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-etc-kubernetes\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-socket-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.070035 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-sys-fs\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069325 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qcx\" (UniqueName: \"kubernetes.io/projected/ab36263f-c154-4f9f-9a8f-10fc38405e52-kube-api-access-n7qcx\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f619d36-116d-4434-bf1e-1aecb4d7fefe-agent-certs\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.069567 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.069618 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret podName:5a8820c7-364d-4f01-90e2-ea9979976e47 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:42.56960065 +0000 UTC m=+3.074243787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret") pod "global-pull-secret-syncer-dt49r" (UID: "5a8820c7-364d-4f01-90e2-ea9979976e47") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.067727 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-iptables-alerter-script\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069694 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-multus-certs\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069696 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-systemd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069743 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f619d36-116d-4434-bf1e-1aecb4d7fefe-konnectivity-ca\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069750 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-sys-fs\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069885 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-slash\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.069946 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-hosts-file\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.070262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-script-lib\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.071120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.070295 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.072085 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.070390 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.072085 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.070391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-socket-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.072396 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.072326 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-etc-tuned\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.073461 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.073436 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f619d36-116d-4434-bf1e-1aecb4d7fefe-agent-certs\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:42.074099 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.074076 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-tmp\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.074184 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.074123 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovn-node-metrics-cert\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.091654 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.091633 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:42.091764 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.091659 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:42.091764 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.091673 2576 projected.go:196] Error preparing data for projected volume kube-api-access-flk52 for pod openshift-network-diagnostics/network-check-target-q74cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:42.091764 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.091760 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52 podName:468096db-c6fe-46bd-b256-05f2132cee4a nodeName:}" failed. No retries permitted until 2026-10-30 17:57:42.591716641 +0000 UTC m=+3.096359780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-flk52" (UniqueName: "kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52") pod "network-check-target-q74cr" (UID: "468096db-c6fe-46bd-b256-05f2132cee4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:42.094316 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.094272 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlqs4\" (UniqueName: \"kubernetes.io/projected/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-kube-api-access-xlqs4\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:42.096605 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.096276 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/66e91a23-8415-48a9-924c-64f752b59272-kube-api-access-59rfw\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.097837 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.097810 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qcx\" (UniqueName: \"kubernetes.io/projected/ab36263f-c154-4f9f-9a8f-10fc38405e52-kube-api-access-n7qcx\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.099686 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.099616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvqb\" (UniqueName: \"kubernetes.io/projected/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-kube-api-access-dgvqb\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.099686 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.099637 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwv6\" (UniqueName: \"kubernetes.io/projected/c598714d-0188-42bd-80e6-d9873dc2ae20-kube-api-access-6dwv6\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.099686 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.099671 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxpz\" (UniqueName: \"kubernetes.io/projected/d79402ab-1f1f-458b-b7c8-3c8209e69efb-kube-api-access-tlxpz\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.099900 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.099705 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zm6w\" (UniqueName: \"kubernetes.io/projected/1bc68151-0620-48d2-a803-e7cb588c5b76-kube-api-access-4zm6w\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:42.099963 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.099923 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7fzn\" (UniqueName: \"kubernetes.io/projected/4f34929e-0b21-4743-af11-1360d39f162b-kube-api-access-p7fzn\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:42.101131 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.101111 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7gv\" (UniqueName: \"kubernetes.io/projected/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-kube-api-access-cg7gv\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:42.176147 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.176113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:42.184453 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.184425 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:57:42.188265 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:42.188238 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f619d36_116d_4434_bf1e_1aecb4d7fefe.slice/crio-e697292bee026df8571cc20abfa9fe1a8186b18c54128a40553fcbbc3fac837d WatchSource:0}: Error finding container e697292bee026df8571cc20abfa9fe1a8186b18c54128a40553fcbbc3fac837d: Status 404 returned error can't find the container with id e697292bee026df8571cc20abfa9fe1a8186b18c54128a40553fcbbc3fac837d Oct 30 17:57:42.194131 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:42.194104 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f34929e_0b21_4743_af11_1360d39f162b.slice/crio-07ea06b86bef3a20ce9115a13047804abe6ed2b3f536afcb024421693455c5aa WatchSource:0}: Error finding container 07ea06b86bef3a20ce9115a13047804abe6ed2b3f536afcb024421693455c5aa: Status 404 returned error can't find the container with id 07ea06b86bef3a20ce9115a13047804abe6ed2b3f536afcb024421693455c5aa Oct 30 17:57:42.194239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.194178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:57:42.200658 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.200496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:57:42.205287 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:42.205249 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e91a23_8415_48a9_924c_64f752b59272.slice/crio-8dc2d5752024ead07a607557ad0a83f6f14cba92379fb9e008325b19a2481f7d WatchSource:0}: Error finding container 8dc2d5752024ead07a607557ad0a83f6f14cba92379fb9e008325b19a2481f7d: Status 404 returned error can't find the container with id 8dc2d5752024ead07a607557ad0a83f6f14cba92379fb9e008325b19a2481f7d Oct 30 17:57:42.212316 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:42.212291 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e4d7d6_f980_4b03_8200_d6c5d8d3dc40.slice/crio-6e212f4ef613c4cf68f2ead0714d8b305f4eaa39361270349e41c22b0cb99416 WatchSource:0}: Error finding container 6e212f4ef613c4cf68f2ead0714d8b305f4eaa39361270349e41c22b0cb99416: Status 404 returned error can't find the container with id 6e212f4ef613c4cf68f2ead0714d8b305f4eaa39361270349e41c22b0cb99416 Oct 30 17:57:42.214218 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.214195 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:57:42.226328 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:42.226300 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc598714d_0188_42bd_80e6_d9873dc2ae20.slice/crio-0654be5cdc8e86bc08fae1da65afa1855395fa4993dee17b4990e536079479a6 WatchSource:0}: Error finding container 0654be5cdc8e86bc08fae1da65afa1855395fa4993dee17b4990e536079479a6: Status 404 returned error can't find the container with id 0654be5cdc8e86bc08fae1da65afa1855395fa4993dee17b4990e536079479a6 Oct 30 17:57:42.236043 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.236006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ch8b8" Oct 30 17:57:42.240211 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.240115 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:57:42.242396 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.242372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:57:42.248252 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:42.247528 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab36263f_c154_4f9f_9a8f_10fc38405e52.slice/crio-65a1ff50dc1d7c32bf2b223fd3cf3bb7daa2be665fdec8fd3cc474af1a0fc815 WatchSource:0}: Error finding container 65a1ff50dc1d7c32bf2b223fd3cf3bb7daa2be665fdec8fd3cc474af1a0fc815: Status 404 returned error can't find the container with id 65a1ff50dc1d7c32bf2b223fd3cf3bb7daa2be665fdec8fd3cc474af1a0fc815 Oct 30 17:57:42.252029 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:42.252002 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79402ab_1f1f_458b_b7c8_3c8209e69efb.slice/crio-ba92e9d064ca58f324001642abb2d2f5ae8fdb9b3acb7feb20f0f7a85ff9c522 WatchSource:0}: Error finding container ba92e9d064ca58f324001642abb2d2f5ae8fdb9b3acb7feb20f0f7a85ff9c522: Status 404 returned error can't find the container with id ba92e9d064ca58f324001642abb2d2f5ae8fdb9b3acb7feb20f0f7a85ff9c522 Oct 30 17:57:42.254964 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:42.254942 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748c9d09_9db4_4e7d_9389_1b74c8e0e2c7.slice/crio-1e0b6fd527e592f71ba6e70227f28f619dcdcb162b4ff7b24db493bec7884b17 WatchSource:0}: Error finding container 1e0b6fd527e592f71ba6e70227f28f619dcdcb162b4ff7b24db493bec7884b17: Status 404 returned error can't find the container with id 1e0b6fd527e592f71ba6e70227f28f619dcdcb162b4ff7b24db493bec7884b17 Oct 30 17:57:42.259159 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.259138 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:57:42.270542 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:57:42.270214 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c4dbcb_db0a_4e7d_a633_3ff9304037d4.slice/crio-a78c392cc5757ee37c6a65b839638dc72ac734944b3661dc542cb126d0044288 WatchSource:0}: Error finding container a78c392cc5757ee37c6a65b839638dc72ac734944b3661dc542cb126d0044288: Status 404 returned error can't find the container with id a78c392cc5757ee37c6a65b839638dc72ac734944b3661dc542cb126d0044288 Oct 30 17:57:42.573648 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.573623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:42.573782 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.573670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:42.573842 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.573786 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:42.573842 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.573786 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:42.573933 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.573856 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:43.573836218 +0000 UTC m=+4.078479374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:42.573933 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.573876 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret podName:5a8820c7-364d-4f01-90e2-ea9979976e47 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:43.573865736 +0000 UTC m=+4.078508855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret") pod "global-pull-secret-syncer-dt49r" (UID: "5a8820c7-364d-4f01-90e2-ea9979976e47") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:42.674720 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.674387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:42.674720 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.674689 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:42.674720 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.674721 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:42.674975 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.674735 2576 projected.go:196] Error preparing data for projected volume kube-api-access-flk52 for pod openshift-network-diagnostics/network-check-target-q74cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:42.674975 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.674804 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52 podName:468096db-c6fe-46bd-b256-05f2132cee4a nodeName:}" failed. No retries permitted until 2026-10-30 17:57:43.674784223 +0000 UTC m=+4.179427362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-flk52" (UniqueName: "kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52") pod "network-check-target-q74cr" (UID: "468096db-c6fe-46bd-b256-05f2132cee4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:42.932620 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.932455 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:52:42 +0000 UTC" deadline="2028-08-05 18:19:29.834444614 +0000 UTC" Oct 30 17:57:42.932620 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.932521 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15480h21m46.901928297s" Oct 30 17:57:42.978089 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.977276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:42.978089 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:42.977411 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:42.988355 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.988297 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" event={"ID":"56c4dbcb-db0a-4e7d-a633-3ff9304037d4","Type":"ContainerStarted","Data":"a78c392cc5757ee37c6a65b839638dc72ac734944b3661dc542cb126d0044288"} Oct 30 17:57:42.991148 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.991118 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" event={"ID":"d79402ab-1f1f-458b-b7c8-3c8209e69efb","Type":"ContainerStarted","Data":"ba92e9d064ca58f324001642abb2d2f5ae8fdb9b3acb7feb20f0f7a85ff9c522"} Oct 30 17:57:42.995951 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:42.995924 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ch8b8" event={"ID":"ab36263f-c154-4f9f-9a8f-10fc38405e52","Type":"ContainerStarted","Data":"65a1ff50dc1d7c32bf2b223fd3cf3bb7daa2be665fdec8fd3cc474af1a0fc815"} Oct 30 17:57:43.005716 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.005691 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" event={"ID":"c598714d-0188-42bd-80e6-d9873dc2ae20","Type":"ContainerStarted","Data":"0654be5cdc8e86bc08fae1da65afa1855395fa4993dee17b4990e536079479a6"} Oct 30 17:57:43.008556 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.008485 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" event={"ID":"66e91a23-8415-48a9-924c-64f752b59272","Type":"ContainerStarted","Data":"8dc2d5752024ead07a607557ad0a83f6f14cba92379fb9e008325b19a2481f7d"} Oct 30 17:57:43.016643 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.016609 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vz7gh" event={"ID":"7f619d36-116d-4434-bf1e-1aecb4d7fefe","Type":"ContainerStarted","Data":"e697292bee026df8571cc20abfa9fe1a8186b18c54128a40553fcbbc3fac837d"} Oct 30 17:57:43.032817 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.032601 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" event={"ID":"46fe009f9a7f834e879e5480aab152d6","Type":"ContainerStarted","Data":"33035195870ba00fd49df006b0f69ed4068c47a0e80b4a2cdbf12e0807f8caa4"} Oct 30 17:57:43.051033 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.050410 2576 generic.go:356] "Generic (PLEG): container finished" podID="67318722ef4c3e2a1b8bc7b38d57c5ad" containerID="e741c9bb968530dc0edbdae04b72a794b5b162deb5c0bec13646df0f86b8399b" exitCode=0 Oct 30 17:57:43.051033 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.050788 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" event={"ID":"67318722ef4c3e2a1b8bc7b38d57c5ad","Type":"ContainerDied","Data":"e741c9bb968530dc0edbdae04b72a794b5b162deb5c0bec13646df0f86b8399b"} Oct 30 17:57:43.060454 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.060428 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j8t4c" event={"ID":"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7","Type":"ContainerStarted","Data":"1e0b6fd527e592f71ba6e70227f28f619dcdcb162b4ff7b24db493bec7884b17"} Oct 30 17:57:43.070908 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.070834 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rv2ks" event={"ID":"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40","Type":"ContainerStarted","Data":"6e212f4ef613c4cf68f2ead0714d8b305f4eaa39361270349e41c22b0cb99416"} Oct 30 17:57:43.080530 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.080462 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wlg8" event={"ID":"4f34929e-0b21-4743-af11-1360d39f162b","Type":"ContainerStarted","Data":"07ea06b86bef3a20ce9115a13047804abe6ed2b3f536afcb024421693455c5aa"} Oct 30 17:57:43.118748 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.117773 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" podStartSLOduration=1.117751673 podStartE2EDuration="1.117751673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:43.07647008 +0000 UTC m=+3.581113225" watchObservedRunningTime="2026-10-30 17:57:43.117751673 +0000 UTC m=+3.622394817" Oct 30 17:57:43.583262 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.582413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:43.583262 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.582513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:43.583262 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.582651 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:43.583262 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.582715 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret podName:5a8820c7-364d-4f01-90e2-ea9979976e47 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:45.582697051 +0000 UTC m=+6.087340186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret") pod "global-pull-secret-syncer-dt49r" (UID: "5a8820c7-364d-4f01-90e2-ea9979976e47") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:43.583262 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.583121 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:43.583262 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.583173 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:45.583156763 +0000 UTC m=+6.087799896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:43.683156 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.683095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:43.683922 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.683386 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:43.683922 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.683416 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:43.683922 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.683430 2576 projected.go:196] Error preparing data for projected volume kube-api-access-flk52 for pod openshift-network-diagnostics/network-check-target-q74cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:43.683922 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.683527 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52 podName:468096db-c6fe-46bd-b256-05f2132cee4a nodeName:}" failed. No retries permitted until 2026-10-30 17:57:45.68347358 +0000 UTC m=+6.188116701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-flk52" (UniqueName: "kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52") pod "network-check-target-q74cr" (UID: "468096db-c6fe-46bd-b256-05f2132cee4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:43.933671 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.933603 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:52:42 +0000 UTC" deadline="2028-05-14 02:26:31.777651614 +0000 UTC" Oct 30 17:57:43.933671 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.933669 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13472h28m47.843987047s" Oct 30 17:57:43.978520 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.978487 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:43.978680 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.978648 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:43.978757 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:43.978732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:43.978837 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:43.978814 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:57:44.094164 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:44.094035 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" event={"ID":"67318722ef4c3e2a1b8bc7b38d57c5ad","Type":"ContainerStarted","Data":"45e0842ad0c9063ea6c307498973e1bba051082e0059aa79fd3594103661bbb8"} Oct 30 17:57:44.975909 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:44.975707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:44.975909 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:44.975844 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:45.103878 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:45.102375 2576 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="4adeba7a8f3b1f39480e52fecacd67106e3df7e524100bae9ba3f40b7b75d499" exitCode=0 Oct 30 17:57:45.103878 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:45.103661 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" event={"ID":"56c4dbcb-db0a-4e7d-a633-3ff9304037d4","Type":"ContainerDied","Data":"4adeba7a8f3b1f39480e52fecacd67106e3df7e524100bae9ba3f40b7b75d499"} Oct 30 17:57:45.171466 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:45.171407 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" podStartSLOduration=3.171388688 podStartE2EDuration="3.171388688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:44.134897488 +0000 UTC m=+4.639540631" watchObservedRunningTime="2026-10-30 17:57:45.171388688 +0000 UTC m=+5.676031830" Oct 30 17:57:45.601582 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:45.601545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:45.601773 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:45.601607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:45.601773 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.601770 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:45.601887 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.601818 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:49.601805465 +0000 UTC m=+10.106448583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:45.602135 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.602112 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:45.602215 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.602152 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret podName:5a8820c7-364d-4f01-90e2-ea9979976e47 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:49.602142696 +0000 UTC m=+10.106785814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret") pod "global-pull-secret-syncer-dt49r" (UID: "5a8820c7-364d-4f01-90e2-ea9979976e47") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:45.702706 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:45.702066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:45.702706 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.702220 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:45.702706 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.702240 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:45.702706 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.702256 2576 projected.go:196] Error preparing data for projected volume kube-api-access-flk52 for pod openshift-network-diagnostics/network-check-target-q74cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:45.702706 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.702330 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52 podName:468096db-c6fe-46bd-b256-05f2132cee4a nodeName:}" failed. No retries permitted until 2026-10-30 17:57:49.702293146 +0000 UTC m=+10.206936265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-flk52" (UniqueName: "kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52") pod "network-check-target-q74cr" (UID: "468096db-c6fe-46bd-b256-05f2132cee4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:45.977555 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:45.977469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:45.978024 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.977649 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:45.978173 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:45.978153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:45.978333 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:45.978304 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:57:46.976875 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:46.976205 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:46.976875 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:46.976418 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:47.111770 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:47.111723 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vz7gh" event={"ID":"7f619d36-116d-4434-bf1e-1aecb4d7fefe","Type":"ContainerStarted","Data":"22b6670fbe359b2fbca346b87612e9275037040caca69982f6340dfa7be59e0e"} Oct 30 17:57:47.662396 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:47.662308 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:47.666458 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:47.662939 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:47.707417 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:47.703818 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vz7gh" podStartSLOduration=3.497849428 podStartE2EDuration="7.70379723s" totalImagesPullingTime="4.205947802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:47.150570633 +0000 UTC m=+7.655213775" watchObservedRunningTime="2026-10-30 17:57:47.70379723 +0000 UTC m=+8.208440375" Oct 30 17:57:47.978656 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:47.978589 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:47.978830 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:47.978731 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:47.978830 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:47.978809 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:47.978944 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:47.978881 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:57:48.119508 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:48.119436 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6wlg8" event={"ID":"4f34929e-0b21-4743-af11-1360d39f162b","Type":"ContainerStarted","Data":"e9baffb47f8c2572e9f7efc69d95f36f22dd2f311d8b5d1c64fa8f81bdfd8f1f"} Oct 30 17:57:48.162419 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:48.161041 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6wlg8" podStartSLOduration=2.823623388 podStartE2EDuration="8.161021419s" totalImagesPullingTime="5.337398031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:48.160440572 +0000 UTC m=+8.665083716" watchObservedRunningTime="2026-10-30 17:57:48.161021419 +0000 UTC m=+8.665664565" Oct 30 17:57:48.975991 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:48.975892 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:48.976220 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:48.976028 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:49.124188 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.124149 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" event={"ID":"c598714d-0188-42bd-80e6-d9873dc2ae20","Type":"ContainerStarted","Data":"51d68afbef678ccbf860998cba369d6c9becade10a3154187bb181995abd4d54"} Oct 30 17:57:49.124749 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.124251 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:57:49.637545 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.637506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:49.637713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.637576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:49.637780 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.637716 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:49.637835 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.637795 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:57.637777327 +0000 UTC m=+18.142420454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:49.638260 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.638238 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:49.638370 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.638320 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret podName:5a8820c7-364d-4f01-90e2-ea9979976e47 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:57.638302699 +0000 UTC m=+18.142945838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret") pod "global-pull-secret-syncer-dt49r" (UID: "5a8820c7-364d-4f01-90e2-ea9979976e47") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:49.738379 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.738349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:49.738601 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.738582 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:49.738601 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.738606 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:49.738758 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.738620 2576 projected.go:196] Error preparing data for projected volume kube-api-access-flk52 for pod openshift-network-diagnostics/network-check-target-q74cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:49.738758 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.738682 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52 podName:468096db-c6fe-46bd-b256-05f2132cee4a nodeName:}" failed. No retries permitted until 2026-10-30 17:57:57.738663797 +0000 UTC m=+18.243306938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-flk52" (UniqueName: "kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52") pod "network-check-target-q74cr" (UID: "468096db-c6fe-46bd-b256-05f2132cee4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:49.765619 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.765590 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:57:49.924481 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.923400 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:57:49.76561564Z","UUID":"e6042273-f1ee-46d7-b183-6fee78c4addb","Handler":null,"Name":"","Endpoint":""} Oct 30 17:57:49.926210 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.926180 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:57:49.926210 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.926213 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:57:49.977507 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.977001 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:49.977507 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.977126 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:57:49.978120 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:49.977940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:49.978120 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:49.978068 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:50.131984 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:50.131953 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" event={"ID":"c598714d-0188-42bd-80e6-d9873dc2ae20","Type":"ContainerStarted","Data":"ecd39e81b0855085c37eb9e9fe64d570f29634e945921ce33d4de3e22200dd0e"} Oct 30 17:57:50.976208 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:50.976050 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:50.976333 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:50.976314 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:51.136479 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:51.136449 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rv2ks" event={"ID":"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40","Type":"ContainerStarted","Data":"c6e1be14e40dbc3f3d2e176315f1227bf2c1aa0ac330f1ff9bfbc3323ee79eda"} Oct 30 17:57:51.139113 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:51.139082 2576 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="9e8495a0f874548a99e2cd72fb5c24a9c38a991dfc14c6fcffb2acea9bff0ac0" exitCode=0 Oct 30 17:57:51.139223 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:51.139181 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" event={"ID":"56c4dbcb-db0a-4e7d-a633-3ff9304037d4","Type":"ContainerDied","Data":"9e8495a0f874548a99e2cd72fb5c24a9c38a991dfc14c6fcffb2acea9bff0ac0"} Oct 30 17:57:51.142374 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:51.142348 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ch8b8" event={"ID":"ab36263f-c154-4f9f-9a8f-10fc38405e52","Type":"ContainerStarted","Data":"421d0777d170ee4606420e6e5581b1639518a53aa84eb37d901477ffe2e6cf6d"} Oct 30 17:57:51.180354 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:51.180287 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rv2ks" podStartSLOduration=1.7207887130000001 podStartE2EDuration="10.180271761s" totalImagesPullingTime="8.459483048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:51.180219372 +0000 UTC m=+11.684862534" watchObservedRunningTime="2026-10-30 17:57:51.180271761 +0000 UTC m=+11.684914902" Oct 30 17:57:51.229084 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:51.228908 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ch8b8" podStartSLOduration=1.847054282 podStartE2EDuration="10.228889363s" totalImagesPullingTime="8.381835081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:51.228431463 +0000 UTC m=+11.733074606" watchObservedRunningTime="2026-10-30 17:57:51.228889363 +0000 UTC m=+11.733532505" Oct 30 17:57:51.978579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:51.978545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:51.978718 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:51.978680 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:51.979069 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:51.979051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:51.979171 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:51.979151 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:57:52.149682 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:52.148787 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" event={"ID":"c598714d-0188-42bd-80e6-d9873dc2ae20","Type":"ContainerStarted","Data":"3eb65c4bd65dc1fe85361cd4ce8725a02aef88fe18afd884fa2631cce50d3f8b"} Oct 30 17:57:52.153281 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:52.153230 2576 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="6024365945f9cb716064178b533f05a0c6d709e025eb02d01f83312a42a45ba5" exitCode=0 Oct 30 17:57:52.153455 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:52.153368 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" event={"ID":"56c4dbcb-db0a-4e7d-a633-3ff9304037d4","Type":"ContainerDied","Data":"6024365945f9cb716064178b533f05a0c6d709e025eb02d01f83312a42a45ba5"} Oct 30 17:57:52.189479 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:52.189420 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" podStartSLOduration=2.770505866 podStartE2EDuration="11.189405118s" totalImagesPullingTime="8.418899252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:52.18880275 +0000 UTC m=+12.693445893" watchObservedRunningTime="2026-10-30 17:57:52.189405118 +0000 UTC m=+12.694048281" Oct 30 17:57:52.976017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:52.975942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:52.976163 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:52.976061 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:53.157224 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:53.157183 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j8t4c" event={"ID":"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7","Type":"ContainerStarted","Data":"6e13e5b04f21ca0abe28e6dddae2d71ec1b9a7d15d02a01ec3016940e0848b78"} Oct 30 17:57:53.193647 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:53.193603 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j8t4c" podStartSLOduration=3.505982903 podStartE2EDuration="12.193588586s" totalImagesPullingTime="8.687605683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:53.193347352 +0000 UTC m=+13.697990496" watchObservedRunningTime="2026-10-30 17:57:53.193588586 +0000 UTC m=+13.698231726" Oct 30 17:57:53.976523 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:53.976442 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:53.976698 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:53.976448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:53.976698 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:53.976589 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:53.976698 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:53.976663 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:57:54.162847 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:54.162805 2576 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="ffde1b0388f82041b371b5e5396324962661d1898eb7a3f36eb79b817f9d9529" exitCode=0 Oct 30 17:57:54.163586 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:54.162898 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" event={"ID":"56c4dbcb-db0a-4e7d-a633-3ff9304037d4","Type":"ContainerDied","Data":"ffde1b0388f82041b371b5e5396324962661d1898eb7a3f36eb79b817f9d9529"} Oct 30 17:57:54.166425 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:54.166049 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" event={"ID":"66e91a23-8415-48a9-924c-64f752b59272","Type":"ContainerStarted","Data":"6298f2240173590b81c51bef7fc7b3c6e2a409535520e62c54b8663f7c77e44d"} Oct 30 17:57:54.253398 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:54.253314 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" podStartSLOduration=2.298371997 podStartE2EDuration="13.253300284s" totalImagesPullingTime="10.954928287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:54.252717564 +0000 UTC m=+14.757360706" watchObservedRunningTime="2026-10-30 17:57:54.253300284 +0000 UTC m=+14.757943422" Oct 30 17:57:54.975902 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:54.975873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:54.976071 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:54.975975 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:55.976752 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:55.976563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:55.977150 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:55.976591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:55.977150 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:55.976834 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:57:55.977150 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:55.976953 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:56.173959 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:56.173925 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" event={"ID":"56c4dbcb-db0a-4e7d-a633-3ff9304037d4","Type":"ContainerStarted","Data":"17b9964111f6c0842c35f80c4751fc81c574460ad498e19b8bfddcce28975def"} Oct 30 17:57:56.976465 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:56.976430 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:56.976695 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:56.976566 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:57.178305 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:57.178277 2576 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="17b9964111f6c0842c35f80c4751fc81c574460ad498e19b8bfddcce28975def" exitCode=0 Oct 30 17:57:57.178689 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:57.178327 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" event={"ID":"56c4dbcb-db0a-4e7d-a633-3ff9304037d4","Type":"ContainerDied","Data":"17b9964111f6c0842c35f80c4751fc81c574460ad498e19b8bfddcce28975def"} Oct 30 17:57:57.701074 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:57.700920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:57.701074 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.701063 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:57.701243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:57.701103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:57.701243 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.701119 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret podName:5a8820c7-364d-4f01-90e2-ea9979976e47 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:13.70110278 +0000 UTC m=+34.205745912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret") pod "global-pull-secret-syncer-dt49r" (UID: "5a8820c7-364d-4f01-90e2-ea9979976e47") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:57:57.701243 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.701184 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:57.701243 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.701237 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:13.701222506 +0000 UTC m=+34.205865636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:57:57.802358 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:57.802327 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:57.802486 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.802444 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:57:57.802486 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.802458 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:57:57.802486 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.802471 2576 projected.go:196] Error preparing data for projected volume kube-api-access-flk52 for pod openshift-network-diagnostics/network-check-target-q74cr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:57.802594 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.802516 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52 podName:468096db-c6fe-46bd-b256-05f2132cee4a nodeName:}" failed. No retries permitted until 2026-10-30 17:58:13.802504263 +0000 UTC m=+34.307147392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-flk52" (UniqueName: "kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52") pod "network-check-target-q74cr" (UID: "468096db-c6fe-46bd-b256-05f2132cee4a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:57:57.975620 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:57.975595 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:57.975731 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.975695 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:57.975792 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:57.975778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:57.975892 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:57.975875 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:57:58.182766 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.182710 2576 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="5a0078c8f530cbe1f0d5871b8e830187e4b95c7e49f279b6aa3d2a8c7c729265" exitCode=0 Oct 30 17:57:58.183126 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.182779 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" event={"ID":"56c4dbcb-db0a-4e7d-a633-3ff9304037d4","Type":"ContainerDied","Data":"5a0078c8f530cbe1f0d5871b8e830187e4b95c7e49f279b6aa3d2a8c7c729265"} Oct 30 17:57:58.185874 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.185856 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" event={"ID":"d79402ab-1f1f-458b-b7c8-3c8209e69efb","Type":"ContainerStarted","Data":"69fb650016c0d68925942bbd6644d40e4d7f683fdcd9c53f9549a8e0281b846c"} Oct 30 17:57:58.185940 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.185908 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" event={"ID":"d79402ab-1f1f-458b-b7c8-3c8209e69efb","Type":"ContainerStarted","Data":"ef85d2ec5dae907a8e9161486e85cabc174296d2b07d90b8c96544606dab58a4"} Oct 30 17:57:58.185940 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.185938 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" event={"ID":"d79402ab-1f1f-458b-b7c8-3c8209e69efb","Type":"ContainerStarted","Data":"ace2cea77b0863fde6f374ade90913e175531063f20e68bd0eadf14fc6e2ec87"} Oct 30 17:57:58.186014 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.185966 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" event={"ID":"d79402ab-1f1f-458b-b7c8-3c8209e69efb","Type":"ContainerStarted","Data":"1579bc899635878c9494f07d8c94aa3cf6410787cd699c7869d204a9462e9bf0"} Oct 30 17:57:58.186014 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.185992 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" event={"ID":"d79402ab-1f1f-458b-b7c8-3c8209e69efb","Type":"ContainerStarted","Data":"e77fe1061aebd618bf3ce6500a86e41e1d0a2f337dfbf8f8d706f7b208dc4486"} Oct 30 17:57:58.186093 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.186018 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" event={"ID":"d79402ab-1f1f-458b-b7c8-3c8209e69efb","Type":"ContainerStarted","Data":"c5079091c6ad693e9ef022c9d4f5c56e6dc59ef96830f2a08e6e075b0b492eb1"} Oct 30 17:57:58.577589 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.577562 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:58.577706 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.577650 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:57:58.580582 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.578781 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:57:58.976397 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:58.976337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:57:58.976515 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:58.976421 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:57:59.190946 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:59.190915 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" event={"ID":"56c4dbcb-db0a-4e7d-a633-3ff9304037d4","Type":"ContainerStarted","Data":"df2408e05ab8cd1f63e76872e7582eb950ef1b8b71d2147ef130ea0f0b3e6e67"} Oct 30 17:57:59.251631 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:59.251536 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p4pv4" podStartSLOduration=6.952176746 podStartE2EDuration="18.251523514s" totalImagesPullingTime="11.299346768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:59.241156372 +0000 UTC m=+19.745799509" watchObservedRunningTime="2026-10-30 17:57:59.251523514 +0000 UTC m=+19.756166651" Oct 30 17:57:59.977002 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:59.976935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:57:59.977151 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:59.977037 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:57:59.977151 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:57:59.977136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:57:59.977278 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:57:59.977241 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:58:00.196172 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:00.196145 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" event={"ID":"d79402ab-1f1f-458b-b7c8-3c8209e69efb","Type":"ContainerStarted","Data":"0c7b470c0656cb1eb7066a9eddc0fd421b7a1d18ffc6c2c0caa6340f28cc3531"} Oct 30 17:58:00.975965 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:00.975942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:00.976083 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:00.976024 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:58:01.975669 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:01.975639 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:01.976375 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:01.975672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:01.976375 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:01.975773 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:58:01.976375 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:01.975865 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:58:02.202785 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:02.202750 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" event={"ID":"d79402ab-1f1f-458b-b7c8-3c8209e69efb","Type":"ContainerStarted","Data":"e5e5539a7e4191654f64d7f5c0ec658fa62606b8d9780cc636163ab7dcd38ae5"} Oct 30 17:58:02.203010 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:02.202991 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:02.218439 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:02.218404 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:02.279915 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:02.279869 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" podStartSLOduration=6.207413687 podStartE2EDuration="21.279857176s" totalImagesPullingTime="15.072443489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:02.269158009 +0000 UTC m=+22.773801125" watchObservedRunningTime="2026-10-30 17:58:02.279857176 +0000 UTC m=+22.784500313" Oct 30 17:58:02.364992 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:02.364832 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:02.976212 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:02.976183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:02.976643 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:02.976291 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:58:03.206119 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:03.206084 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:03.222571 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:03.222549 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:03.921306 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:03.921257 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dt49r"] Oct 30 17:58:03.921459 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:03.921432 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:03.921537 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:03.921520 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:58:03.923698 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:03.923674 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q74cr"] Oct 30 17:58:03.923946 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:03.923920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:03.924065 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:03.924021 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:58:03.925011 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:03.924981 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hk9ss"] Oct 30 17:58:03.925126 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:03.925079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:03.925213 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:03.925189 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:58:04.975516 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:04.975481 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:04.975516 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:04.975502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:04.975910 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:04.975583 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:58:04.975910 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:04.975692 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:58:05.976015 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:05.975987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:05.976355 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:05.976096 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:58:06.975886 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:06.975858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:06.976036 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:06.975859 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:06.976036 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:06.975950 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:58:06.976036 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:06.976021 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:58:07.976503 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:07.976473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:07.976833 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:07.976614 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:58:08.975995 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:08.975962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:08.975995 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:08.975979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:08.976170 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:08.976060 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q74cr" podUID="468096db-c6fe-46bd-b256-05f2132cee4a" Oct 30 17:58:08.976170 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:08.976122 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dt49r" podUID="5a8820c7-364d-4f01-90e2-ea9979976e47" Oct 30 17:58:09.977002 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:09.976973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:09.977486 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:09.977105 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk9ss" podUID="1bc68151-0620-48d2-a803-e7cb588c5b76" Oct 30 17:58:10.302138 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.302009 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeReady" Oct 30 17:58:10.302297 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.302216 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:58:10.356934 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.356911 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qnth8"] Oct 30 17:58:10.359840 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.359822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qnth8" Oct 30 17:58:10.363138 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.363092 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mk86v"] Oct 30 17:58:10.365872 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.365853 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6dk47"] Oct 30 17:58:10.366375 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.366351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:10.373329 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.373311 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6cmjc"] Oct 30 17:58:10.373430 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.373419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6dk47" Oct 30 17:58:10.376807 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.376791 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j"] Oct 30 17:58:10.376936 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.376918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" Oct 30 17:58:10.379229 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.379213 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zg24h"] Oct 30 17:58:10.379368 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.379349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.382213 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.382197 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55dcc4fcc6-4mw7c"] Oct 30 17:58:10.382361 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.382345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.386385 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.386365 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-ff9667845-fjrpd"] Oct 30 17:58:10.386552 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.386537 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.387315 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.387294 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-c6ggl\"" Oct 30 17:58:10.387697 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.387682 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 17:58:10.389604 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.389589 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74f5c5549c-bgqzt"] Oct 30 17:58:10.389712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.389696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.392962 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.392948 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gxwj6"] Oct 30 17:58:10.393088 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.393072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.393833 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.393817 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.393881 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.393817 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.393990 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.393976 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 17:58:10.396238 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.396220 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz"] Oct 30 17:58:10.396366 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.396350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.399455 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.399435 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv"] Oct 30 17:58:10.399542 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.399518 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.402762 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.402745 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k"] Oct 30 17:58:10.402860 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.402845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:10.406355 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.406336 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m"] Oct 30 17:58:10.406644 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.406514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" Oct 30 17:58:10.412645 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.412628 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm"] Oct 30 17:58:10.412742 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.412729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:10.415842 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.415823 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qnth8"] Oct 30 17:58:10.415935 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.415860 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j"] Oct 30 17:58:10.415935 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.415883 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6dk47"] Oct 30 17:58:10.415935 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.415904 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zg24h"] Oct 30 17:58:10.416084 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.415970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" Oct 30 17:58:10.416232 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.415974 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mk86v"] Oct 30 17:58:10.416320 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.416238 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ff9667845-fjrpd"] Oct 30 17:58:10.424197 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.424177 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55dcc4fcc6-4mw7c"] Oct 30 17:58:10.435092 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.435068 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r94b8"] Oct 30 17:58:10.448478 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.448453 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s2zkr"] Oct 30 17:58:10.449516 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.449495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.453546 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.453529 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m"] Oct 30 17:58:10.453665 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.453644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:10.453763 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.453668 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm"] Oct 30 17:58:10.453763 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.453729 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv"] Oct 30 17:58:10.455181 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.455167 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz"] Oct 30 17:58:10.457520 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.457476 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gxwj6"] Oct 30 17:58:10.458629 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.458610 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74f5c5549c-bgqzt"] Oct 30 17:58:10.464952 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.464931 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6cmjc"] Oct 30 17:58:10.483976 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.483958 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r94b8"] Oct 30 17:58:10.492926 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.492901 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:58:10.493324 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.493239 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:58:10.493554 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.493537 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 17:58:10.493619 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.493600 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 17:58:10.493724 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.493709 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:58:10.493850 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.493835 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:58:10.493894 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.493836 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdnw2\"" Oct 30 17:58:10.494389 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.494371 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 17:58:10.494533 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.494518 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 17:58:10.494610 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.494589 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 17:58:10.497239 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.497351 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzmq\" (UniqueName: \"kubernetes.io/projected/d8c35677-9ec2-4c68-938f-a9cfcc872f79-kube-api-access-pvzmq\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:10.497351 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:10.497351 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp5lz\" (UniqueName: \"kubernetes.io/projected/5a311faf-5bcd-47ab-b102-62aa0abf168c-kube-api-access-mp5lz\") pod \"kas-connection-checker-685895c7c9-qnth8\" (UID: \"5a311faf-5bcd-47ab-b102-62aa0abf168c\") " pod="kube-system/kas-connection-checker-685895c7c9-qnth8" Oct 30 17:58:10.497494 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.497494 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.497494 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.497494 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-stats-auth\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.497494 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/5b5e199e-2896-450f-9e73-838fb7e12e48-kube-api-access-k6zgv\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.497494 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.497494 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9jp\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fc78\" (UniqueName: \"kubernetes.io/projected/df983796-b36d-40f0-ae14-93ba96e518e3-kube-api-access-6fc78\") pod \"network-check-source-6f78d59f99-rrfpm\" (UID: \"df983796-b36d-40f0-ae14-93ba96e518e3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-snapshots\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mklf\" (UniqueName: \"kubernetes.io/projected/874e6413-424d-4443-80fc-f9e4d0a63b6c-kube-api-access-9mklf\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b810f479-5f8c-4acc-89de-a41d288c8e2c-serving-cert\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b810f479-5f8c-4acc-89de-a41d288c8e2c-config\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.497810 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hrbz\" (UniqueName: \"kubernetes.io/projected/0bfbeb5b-8284-4324-a8d0-95d95d23dfc6-kube-api-access-7hrbz\") pod \"volume-data-source-validator-58c9f5f46d-lkc5k\" (UID: \"0bfbeb5b-8284-4324-a8d0-95d95d23dfc6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-tmp\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-default-certificate\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.497978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnqk\" (UniqueName: \"kubernetes.io/projected/b810f479-5f8c-4acc-89de-a41d288c8e2c-kube-api-access-hrnqk\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498090 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-serving-cert\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498132 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.498201 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-config\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/874e6413-424d-4443-80fc-f9e4d0a63b6c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498189 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz7ww\" (UniqueName: \"kubernetes.io/projected/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-kube-api-access-cz7ww\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g8s9\" (UniqueName: \"kubernetes.io/projected/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-kube-api-access-5g8s9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4ss\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8ms\" (UniqueName: \"kubernetes.io/projected/f11682e3-9202-425f-af04-da7b7c2351be-kube-api-access-tj8ms\") pod \"kas-connection-checker-685895c7c9-6cmjc\" (UID: \"f11682e3-9202-425f-af04-da7b7c2351be\") " pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/476cffce-abe2-409d-9b4a-792af41d8f1c-serving-cert\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lhrg\" (UniqueName: \"kubernetes.io/projected/476cffce-abe2-409d-9b4a-792af41d8f1c-kube-api-access-4lhrg\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.498787 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.498697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlzws\" (UniqueName: \"kubernetes.io/projected/50098568-1891-40d1-886a-9065582c2467-kube-api-access-hlzws\") pod \"kas-connection-checker-685895c7c9-6dk47\" (UID: \"50098568-1891-40d1-886a-9065582c2467\") " pod="kube-system/kas-connection-checker-685895c7c9-6dk47" Oct 30 17:58:10.502721 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.502671 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 17:58:10.502947 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.502927 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.503038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.502962 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 17:58:10.503038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.502994 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9dlsp\"" Oct 30 17:58:10.503038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503008 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 17:58:10.503038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503010 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 17:58:10.503197 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503024 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8mdkh\"" Oct 30 17:58:10.503197 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503049 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:58:10.503197 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503050 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.503197 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503139 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.503374 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503311 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nfj27\"" Oct 30 17:58:10.503374 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503315 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 17:58:10.503462 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503390 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:58:10.503462 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503416 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qgvl5\"" Oct 30 17:58:10.503580 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503550 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.503580 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503551 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.503740 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503615 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-swnsr\"" Oct 30 17:58:10.503740 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503620 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 17:58:10.503740 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503696 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dltg8\"" Oct 30 17:58:10.503740 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503703 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.503740 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503730 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503748 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503777 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503774 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503799 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503803 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503810 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503830 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503872 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503933 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zdwtx\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503954 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hhtgn\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.503934 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pn5pr\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504001 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cv2gk\"" Oct 30 17:58:10.503979 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504009 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504056 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504062 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zdsf4\"" Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504101 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504310 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504390 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504418 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k"] Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504431 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504483 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504514 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.504712 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.504570 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:10.505558 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.505544 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s2zkr"] Oct 30 17:58:10.515380 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.515364 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:58:10.525633 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.525610 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 17:58:10.525721 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.525619 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:58:10.525721 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.525683 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 17:58:10.599931 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.599876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.599931 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.599911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/529241e2-eb4e-4412-ad1f-37dee4aa9b65-config-volume\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.600083 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.599963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.600083 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.599990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.600083 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.600083 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529241e2-eb4e-4412-ad1f-37dee4aa9b65-tmp-dir\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.600381 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnqk\" (UniqueName: \"kubernetes.io/projected/b810f479-5f8c-4acc-89de-a41d288c8e2c-kube-api-access-hrnqk\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.600542 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.600609 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600565 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-serving-cert\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.600609 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:10.600714 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.600714 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600661 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.600714 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-config\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.600853 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/874e6413-424d-4443-80fc-f9e4d0a63b6c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:10.600853 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:10.600853 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600813 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrhb\" (UniqueName: \"kubernetes.io/projected/7caaae72-bd7d-44b5-902e-099da2ef429b-kube-api-access-fwrhb\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:10.600853 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7ww\" (UniqueName: \"kubernetes.io/projected/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-kube-api-access-cz7ww\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.601051 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.601051 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.601421 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.601397 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.601656 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.601638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.601891 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.601876 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:11.101849416 +0000 UTC m=+31.606492549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:10.601983 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.601958 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-config\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.602353 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.602321 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/874e6413-424d-4443-80fc-f9e4d0a63b6c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:10.602950 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.602922 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.603065 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:10.603220 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.603203 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:58:10.603304 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.600954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8s9\" (UniqueName: \"kubernetes.io/projected/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-kube-api-access-5g8s9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.603505 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls podName:d8c35677-9ec2-4c68-938f-a9cfcc872f79 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:11.103484026 +0000 UTC m=+31.608127161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c2d9m" (UID: "d8c35677-9ec2-4c68-938f-a9cfcc872f79") : secret "samples-operator-tls" not found Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603527 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4ss\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8ms\" (UniqueName: \"kubernetes.io/projected/f11682e3-9202-425f-af04-da7b7c2351be-kube-api-access-tj8ms\") pod \"kas-connection-checker-685895c7c9-6cmjc\" (UID: \"f11682e3-9202-425f-af04-da7b7c2351be\") " pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/476cffce-abe2-409d-9b4a-792af41d8f1c-serving-cert\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhrg\" (UniqueName: \"kubernetes.io/projected/476cffce-abe2-409d-9b4a-792af41d8f1c-kube-api-access-4lhrg\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzws\" (UniqueName: \"kubernetes.io/projected/50098568-1891-40d1-886a-9065582c2467-kube-api-access-hlzws\") pod \"kas-connection-checker-685895c7c9-6dk47\" (UID: \"50098568-1891-40d1-886a-9065582c2467\") " pod="kube-system/kas-connection-checker-685895c7c9-6dk47" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603831 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzmq\" (UniqueName: \"kubernetes.io/projected/d8c35677-9ec2-4c68-938f-a9cfcc872f79-kube-api-access-pvzmq\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp5lz\" (UniqueName: \"kubernetes.io/projected/5a311faf-5bcd-47ab-b102-62aa0abf168c-kube-api-access-mp5lz\") pod \"kas-connection-checker-685895c7c9-qnth8\" (UID: \"5a311faf-5bcd-47ab-b102-62aa0abf168c\") " pod="kube-system/kas-connection-checker-685895c7c9-qnth8" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.603980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.606579 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-stats-auth\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7caaae72-bd7d-44b5-902e-099da2ef429b-cert\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/5b5e199e-2896-450f-9e73-838fb7e12e48-kube-api-access-k6zgv\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9jp\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fc78\" (UniqueName: \"kubernetes.io/projected/df983796-b36d-40f0-ae14-93ba96e518e3-kube-api-access-6fc78\") pod \"network-check-source-6f78d59f99-rrfpm\" (UID: \"df983796-b36d-40f0-ae14-93ba96e518e3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnl8d\" (UniqueName: \"kubernetes.io/projected/529241e2-eb4e-4412-ad1f-37dee4aa9b65-kube-api-access-bnl8d\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-snapshots\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.610394 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mklf\" (UniqueName: \"kubernetes.io/projected/874e6413-424d-4443-80fc-f9e4d0a63b6c-kube-api-access-9mklf\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b810f479-5f8c-4acc-89de-a41d288c8e2c-serving-cert\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b810f479-5f8c-4acc-89de-a41d288c8e2c-config\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.604735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.605106 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.605811 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-serving-cert\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.605835 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.605883 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.605894 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55dcc4fcc6-4mw7c: secret "image-registry-tls" not found Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.605938 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls podName:056042ce-b312-4d54-9f83-15cd37aa97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:11.105925585 +0000 UTC m=+31.610568701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls") pod "image-registry-55dcc4fcc6-4mw7c" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce") : secret "image-registry-tls" not found Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.606046 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-snapshots\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.606394 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.607046 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.607116 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls podName:874e6413-424d-4443-80fc-f9e4d0a63b6c nodeName:}" failed. No retries permitted until 2026-10-30 17:58:11.107091632 +0000 UTC m=+31.611734765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8g5tv" (UID: "874e6413-424d-4443-80fc-f9e4d0a63b6c") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.607196 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.607591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hrbz\" (UniqueName: \"kubernetes.io/projected/0bfbeb5b-8284-4324-a8d0-95d95d23dfc6-kube-api-access-7hrbz\") pod \"volume-data-source-validator-58c9f5f46d-lkc5k\" (UID: \"0bfbeb5b-8284-4324-a8d0-95d95d23dfc6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" Oct 30 17:58:10.611243 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.607628 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b810f479-5f8c-4acc-89de-a41d288c8e2c-config\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.607639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.607675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-tmp\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.607707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.608317 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.608814 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.608992 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.609033 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-tmp\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.609223 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.609238 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5c5549c-bgqzt: secret "image-registry-tls" not found Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.609312 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls podName:93028940-79f3-45fa-be9b-6ddd8dab5633 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:11.109296384 +0000 UTC m=+31.613939510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls") pod "image-registry-74f5c5549c-bgqzt" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633") : secret "image-registry-tls" not found Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.609394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-default-certificate\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.609456 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.609621 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/476cffce-abe2-409d-9b4a-792af41d8f1c-serving-cert\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.609866 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-stats-auth\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.610637 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.610654 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:10.612017 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.610693 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:58:10.612745 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.610754 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:11.110739188 +0000 UTC m=+31.615382319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : secret "router-metrics-certs-default" not found Oct 30 17:58:10.612745 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.610802 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:11.110792111 +0000 UTC m=+31.615435255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:10.612745 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.611596 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b810f479-5f8c-4acc-89de-a41d288c8e2c-serving-cert\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.612745 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.612046 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-default-certificate\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.612927 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.612903 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.613770 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.613753 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.625653 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.625632 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8s9\" (UniqueName: \"kubernetes.io/projected/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-kube-api-access-5g8s9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.634812 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.634785 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7ww\" (UniqueName: \"kubernetes.io/projected/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-kube-api-access-cz7ww\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.635324 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.635254 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhrg\" (UniqueName: \"kubernetes.io/projected/476cffce-abe2-409d-9b4a-792af41d8f1c-kube-api-access-4lhrg\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.635878 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.635851 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4ss\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.636727 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.636685 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:10.636801 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.636728 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8ms\" (UniqueName: \"kubernetes.io/projected/f11682e3-9202-425f-af04-da7b7c2351be-kube-api-access-tj8ms\") pod \"kas-connection-checker-685895c7c9-6cmjc\" (UID: \"f11682e3-9202-425f-af04-da7b7c2351be\") " pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" Oct 30 17:58:10.636873 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.636854 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9jp\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.636919 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.636901 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mklf\" (UniqueName: \"kubernetes.io/projected/874e6413-424d-4443-80fc-f9e4d0a63b6c-kube-api-access-9mklf\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:10.637651 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.637634 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzws\" (UniqueName: \"kubernetes.io/projected/50098568-1891-40d1-886a-9065582c2467-kube-api-access-hlzws\") pod \"kas-connection-checker-685895c7c9-6dk47\" (UID: \"50098568-1891-40d1-886a-9065582c2467\") " pod="kube-system/kas-connection-checker-685895c7c9-6dk47" Oct 30 17:58:10.638934 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.638915 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/5b5e199e-2896-450f-9e73-838fb7e12e48-kube-api-access-k6zgv\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:10.640208 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.640179 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hrbz\" (UniqueName: \"kubernetes.io/projected/0bfbeb5b-8284-4324-a8d0-95d95d23dfc6-kube-api-access-7hrbz\") pod \"volume-data-source-validator-58c9f5f46d-lkc5k\" (UID: \"0bfbeb5b-8284-4324-a8d0-95d95d23dfc6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" Oct 30 17:58:10.640756 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.640735 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzmq\" (UniqueName: \"kubernetes.io/projected/d8c35677-9ec2-4c68-938f-a9cfcc872f79-kube-api-access-pvzmq\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:10.640971 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.640950 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp5lz\" (UniqueName: \"kubernetes.io/projected/5a311faf-5bcd-47ab-b102-62aa0abf168c-kube-api-access-mp5lz\") pod \"kas-connection-checker-685895c7c9-qnth8\" (UID: \"5a311faf-5bcd-47ab-b102-62aa0abf168c\") " pod="kube-system/kas-connection-checker-685895c7c9-qnth8" Oct 30 17:58:10.641121 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.641097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:10.641552 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.641532 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnqk\" (UniqueName: \"kubernetes.io/projected/b810f479-5f8c-4acc-89de-a41d288c8e2c-kube-api-access-hrnqk\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.642033 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.642017 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fc78\" (UniqueName: \"kubernetes.io/projected/df983796-b36d-40f0-ae14-93ba96e518e3-kube-api-access-6fc78\") pod \"network-check-source-6f78d59f99-rrfpm\" (UID: \"df983796-b36d-40f0-ae14-93ba96e518e3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" Oct 30 17:58:10.676426 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.676402 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-qnth8" Oct 30 17:58:10.690364 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.690351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6dk47" Oct 30 17:58:10.696964 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.696950 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" Oct 30 17:58:10.702636 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.702603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:10.710312 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.710293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:10.711713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.710803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.711713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.710855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrhb\" (UniqueName: \"kubernetes.io/projected/7caaae72-bd7d-44b5-902e-099da2ef429b-kube-api-access-fwrhb\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:10.711713 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.710977 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:10.711713 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:10.711043 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls podName:529241e2-eb4e-4412-ad1f-37dee4aa9b65 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:11.211023452 +0000 UTC m=+31.715666568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls") pod "dns-default-r94b8" (UID: "529241e2-eb4e-4412-ad1f-37dee4aa9b65") : secret "dns-default-metrics-tls" not found Oct 30 17:58:10.711713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.710980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7caaae72-bd7d-44b5-902e-099da2ef429b-cert\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:10.711713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.711124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnl8d\" (UniqueName: \"kubernetes.io/projected/529241e2-eb4e-4412-ad1f-37dee4aa9b65-kube-api-access-bnl8d\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.711713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.711211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/529241e2-eb4e-4412-ad1f-37dee4aa9b65-config-volume\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.711713 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.711244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529241e2-eb4e-4412-ad1f-37dee4aa9b65-tmp-dir\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.712165 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.712040 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/529241e2-eb4e-4412-ad1f-37dee4aa9b65-config-volume\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.712498 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.712471 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529241e2-eb4e-4412-ad1f-37dee4aa9b65-tmp-dir\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.716188 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.716163 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7caaae72-bd7d-44b5-902e-099da2ef429b-cert\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:10.737800 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.734308 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnl8d\" (UniqueName: \"kubernetes.io/projected/529241e2-eb4e-4412-ad1f-37dee4aa9b65-kube-api-access-bnl8d\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:10.737800 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.734564 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrhb\" (UniqueName: \"kubernetes.io/projected/7caaae72-bd7d-44b5-902e-099da2ef429b-kube-api-access-fwrhb\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:10.745136 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.745105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:10.755624 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.755555 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:10.777974 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.769185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" Oct 30 17:58:10.781845 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.781796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" Oct 30 17:58:10.792199 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.792156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:10.976520 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.976496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:10.976727 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.976708 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:10.990938 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.990912 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:58:10.991213 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.990954 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2xlsx\"" Oct 30 17:58:10.991213 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:10.990987 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xl6hf\"" Oct 30 17:58:11.013829 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.013806 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6cmjc"] Oct 30 17:58:11.016770 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.016745 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-qnth8"] Oct 30 17:58:11.018054 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:58:11.018031 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11682e3_9202_425f_af04_da7b7c2351be.slice/crio-2558988e914b7ff0ecf6b2ad7c05113d298e2106d7f045367df2267c79f09499 WatchSource:0}: Error finding container 2558988e914b7ff0ecf6b2ad7c05113d298e2106d7f045367df2267c79f09499: Status 404 returned error can't find the container with id 2558988e914b7ff0ecf6b2ad7c05113d298e2106d7f045367df2267c79f09499 Oct 30 17:58:11.020624 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:58:11.020606 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a311faf_5bcd_47ab_b102_62aa0abf168c.slice/crio-96dd8351e8c09ab7b93fdf0d7494b9a33c1bd4c0b95c281ef9a60055a8c7341f WatchSource:0}: Error finding container 96dd8351e8c09ab7b93fdf0d7494b9a33c1bd4c0b95c281ef9a60055a8c7341f: Status 404 returned error can't find the container with id 96dd8351e8c09ab7b93fdf0d7494b9a33c1bd4c0b95c281ef9a60055a8c7341f Oct 30 17:58:11.027460 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.027443 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6dk47"] Oct 30 17:58:11.031673 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:58:11.031653 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50098568_1891_40d1_886a_9065582c2467.slice/crio-abe3cbfd004f98d7f1a99e6dfda84ed966a56a5b4b4252334ff3f9b8fd6af697 WatchSource:0}: Error finding container abe3cbfd004f98d7f1a99e6dfda84ed966a56a5b4b4252334ff3f9b8fd6af697: Status 404 returned error can't find the container with id abe3cbfd004f98d7f1a99e6dfda84ed966a56a5b4b4252334ff3f9b8fd6af697 Oct 30 17:58:11.060293 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.060133 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zg24h"] Oct 30 17:58:11.063089 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.062740 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j"] Oct 30 17:58:11.069803 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:58:11.069607 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476cffce_abe2_409d_9b4a_792af41d8f1c.slice/crio-b518a841587144a85ff55bd0dfc07e01e250dec1d1cc732ba4299f848eaf0ea6 WatchSource:0}: Error finding container b518a841587144a85ff55bd0dfc07e01e250dec1d1cc732ba4299f848eaf0ea6: Status 404 returned error can't find the container with id b518a841587144a85ff55bd0dfc07e01e250dec1d1cc732ba4299f848eaf0ea6 Oct 30 17:58:11.116471 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.116398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:11.116612 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.116475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:11.116612 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.116543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:11.116612 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.116578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:11.116773 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.116628 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:11.116773 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.116660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:11.116773 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.116709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:11.116921 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.116853 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:11.116921 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.116868 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5c5549c-bgqzt: secret "image-registry-tls" not found Oct 30 17:58:11.117019 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.116923 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls podName:93028940-79f3-45fa-be9b-6ddd8dab5633 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:12.116904281 +0000 UTC m=+32.621547401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls") pod "image-registry-74f5c5549c-bgqzt" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633") : secret "image-registry-tls" not found Oct 30 17:58:11.117306 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117285 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:58:11.117392 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117343 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls podName:d8c35677-9ec2-4c68-938f-a9cfcc872f79 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:12.117327931 +0000 UTC m=+32.621971047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c2d9m" (UID: "d8c35677-9ec2-4c68-938f-a9cfcc872f79") : secret "samples-operator-tls" not found Oct 30 17:58:11.117448 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117420 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:12.117405585 +0000 UTC m=+32.622048702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:11.117507 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117480 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:11.117560 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117512 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls podName:874e6413-424d-4443-80fc-f9e4d0a63b6c nodeName:}" failed. No retries permitted until 2026-10-30 17:58:12.117501307 +0000 UTC m=+32.622144426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8g5tv" (UID: "874e6413-424d-4443-80fc-f9e4d0a63b6c") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:11.117613 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117571 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:11.117613 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117582 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55dcc4fcc6-4mw7c: secret "image-registry-tls" not found Oct 30 17:58:11.117613 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117612 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls podName:056042ce-b312-4d54-9f83-15cd37aa97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:12.117602632 +0000 UTC m=+32.622245747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls") pod "image-registry-55dcc4fcc6-4mw7c" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce") : secret "image-registry-tls" not found Oct 30 17:58:11.117753 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117668 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:11.117753 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117702 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:12.117691287 +0000 UTC m=+32.622334403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:11.117853 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117788 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:58:11.117901 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.117860 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:12.117849328 +0000 UTC m=+32.622492447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : secret "router-metrics-certs-default" not found Oct 30 17:58:11.122581 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.122509 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gxwj6"] Oct 30 17:58:11.127745 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:58:11.127718 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21caa3cf_ca4f_424a_9f40_6f91a7b176c3.slice/crio-72430c9337e74812ba6b2ad90d827aa84f53a792941dbda4e33fd28e92f264fa WatchSource:0}: Error finding container 72430c9337e74812ba6b2ad90d827aa84f53a792941dbda4e33fd28e92f264fa: Status 404 returned error can't find the container with id 72430c9337e74812ba6b2ad90d827aa84f53a792941dbda4e33fd28e92f264fa Oct 30 17:58:11.140764 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.140689 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz"] Oct 30 17:58:11.147628 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.147607 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s2zkr"] Oct 30 17:58:11.149439 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:58:11.149420 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb810f479_5f8c_4acc_89de_a41d288c8e2c.slice/crio-4606c18ee91f4eddbb933f9a86249a4e84f54314c3523d8f76e1db8321eb2f5a WatchSource:0}: Error finding container 4606c18ee91f4eddbb933f9a86249a4e84f54314c3523d8f76e1db8321eb2f5a: Status 404 returned error can't find the container with id 4606c18ee91f4eddbb933f9a86249a4e84f54314c3523d8f76e1db8321eb2f5a Oct 30 17:58:11.209589 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.209563 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm"] Oct 30 17:58:11.213867 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.213793 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k"] Oct 30 17:58:11.217148 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.217109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:11.217257 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.217241 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:11.217334 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:11.217322 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls podName:529241e2-eb4e-4412-ad1f-37dee4aa9b65 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:12.217305509 +0000 UTC m=+32.721948625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls") pod "dns-default-r94b8" (UID: "529241e2-eb4e-4412-ad1f-37dee4aa9b65") : secret "dns-default-metrics-tls" not found Oct 30 17:58:11.221210 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.221189 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" event={"ID":"b810f479-5f8c-4acc-89de-a41d288c8e2c","Type":"ContainerStarted","Data":"4606c18ee91f4eddbb933f9a86249a4e84f54314c3523d8f76e1db8321eb2f5a"} Oct 30 17:58:11.222646 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.222626 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qnth8" event={"ID":"5a311faf-5bcd-47ab-b102-62aa0abf168c","Type":"ContainerStarted","Data":"31da0890e3b63acb97e7429cebc04dc25c7424771758b42e7809908fa23a2cca"} Oct 30 17:58:11.222726 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.222657 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-qnth8" event={"ID":"5a311faf-5bcd-47ab-b102-62aa0abf168c","Type":"ContainerStarted","Data":"96dd8351e8c09ab7b93fdf0d7494b9a33c1bd4c0b95c281ef9a60055a8c7341f"} Oct 30 17:58:11.223615 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.223589 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zg24h" event={"ID":"476cffce-abe2-409d-9b4a-792af41d8f1c","Type":"ContainerStarted","Data":"b518a841587144a85ff55bd0dfc07e01e250dec1d1cc732ba4299f848eaf0ea6"} Oct 30 17:58:11.224551 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.224535 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" event={"ID":"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a","Type":"ContainerStarted","Data":"fa3a9cce436bca035669a359cef965751407914479af843ddae0087d5a983ecd"} Oct 30 17:58:11.225700 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.225681 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6dk47" event={"ID":"50098568-1891-40d1-886a-9065582c2467","Type":"ContainerStarted","Data":"566eece893c0da2a81b4c8a85e040d98998f9c8bac9041893cd9ecc91b5e1f54"} Oct 30 17:58:11.225778 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.225713 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6dk47" event={"ID":"50098568-1891-40d1-886a-9065582c2467","Type":"ContainerStarted","Data":"abe3cbfd004f98d7f1a99e6dfda84ed966a56a5b4b4252334ff3f9b8fd6af697"} Oct 30 17:58:11.226818 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.226787 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s2zkr" event={"ID":"7caaae72-bd7d-44b5-902e-099da2ef429b","Type":"ContainerStarted","Data":"95855e4c59040c66b37ef72a54cb9de5487c85997d619f22c84accec198ab5be"} Oct 30 17:58:11.229908 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.229884 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" event={"ID":"21caa3cf-ca4f-424a-9f40-6f91a7b176c3","Type":"ContainerStarted","Data":"72430c9337e74812ba6b2ad90d827aa84f53a792941dbda4e33fd28e92f264fa"} Oct 30 17:58:11.230473 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:58:11.230454 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfbeb5b_8284_4324_a8d0_95d95d23dfc6.slice/crio-fc6bbf56c484f44ef737360c94843ce9eb85de60ec726eeb3d15cfa53a262413 WatchSource:0}: Error finding container fc6bbf56c484f44ef737360c94843ce9eb85de60ec726eeb3d15cfa53a262413: Status 404 returned error can't find the container with id fc6bbf56c484f44ef737360c94843ce9eb85de60ec726eeb3d15cfa53a262413 Oct 30 17:58:11.233408 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.233329 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" event={"ID":"f11682e3-9202-425f-af04-da7b7c2351be","Type":"ContainerStarted","Data":"6c091743a2556ce7911dc7c02b8136b29539f8dc5cbab5ed351fd38ed1240c97"} Oct 30 17:58:11.233408 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.233371 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" event={"ID":"f11682e3-9202-425f-af04-da7b7c2351be","Type":"ContainerStarted","Data":"2558988e914b7ff0ecf6b2ad7c05113d298e2106d7f045367df2267c79f09499"} Oct 30 17:58:11.979366 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.979333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:11.992963 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.992926 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:58:11.993658 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:11.993589 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6fpb9\"" Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.126396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.126449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.126498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.126523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.126563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.126611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.126677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.126814 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.126795766 +0000 UTC m=+34.631438882 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127180 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127233 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls podName:874e6413-424d-4443-80fc-f9e4d0a63b6c nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.12721726 +0000 UTC m=+34.631860380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8g5tv" (UID: "874e6413-424d-4443-80fc-f9e4d0a63b6c") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127315 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127328 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55dcc4fcc6-4mw7c: secret "image-registry-tls" not found Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127362 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls podName:056042ce-b312-4d54-9f83-15cd37aa97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.127351342 +0000 UTC m=+34.631994461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls") pod "image-registry-55dcc4fcc6-4mw7c" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce") : secret "image-registry-tls" not found Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127416 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:12.128038 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127448 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.127437672 +0000 UTC m=+34.632080794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:12.128961 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127499 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:58:12.128961 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127528 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.127518262 +0000 UTC m=+34.632161380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : secret "router-metrics-certs-default" not found Oct 30 17:58:12.128961 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127570 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:12.128961 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127576 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5c5549c-bgqzt: secret "image-registry-tls" not found Oct 30 17:58:12.128961 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127595 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls podName:93028940-79f3-45fa-be9b-6ddd8dab5633 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.127588633 +0000 UTC m=+34.632231749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls") pod "image-registry-74f5c5549c-bgqzt" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633") : secret "image-registry-tls" not found Oct 30 17:58:12.128961 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127645 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:58:12.128961 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.127675 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls podName:d8c35677-9ec2-4c68-938f-a9cfcc872f79 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.127665936 +0000 UTC m=+34.632309054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c2d9m" (UID: "d8c35677-9ec2-4c68-938f-a9cfcc872f79") : secret "samples-operator-tls" not found Oct 30 17:58:12.227447 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.227412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:12.227636 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.227621 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:12.227692 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:12.227676 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls podName:529241e2-eb4e-4412-ad1f-37dee4aa9b65 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:14.227660599 +0000 UTC m=+34.732303722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls") pod "dns-default-r94b8" (UID: "529241e2-eb4e-4412-ad1f-37dee4aa9b65") : secret "dns-default-metrics-tls" not found Oct 30 17:58:12.246067 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.245936 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" event={"ID":"0bfbeb5b-8284-4324-a8d0-95d95d23dfc6","Type":"ContainerStarted","Data":"fc6bbf56c484f44ef737360c94843ce9eb85de60ec726eeb3d15cfa53a262413"} Oct 30 17:58:12.248662 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.248342 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" event={"ID":"df983796-b36d-40f0-ae14-93ba96e518e3","Type":"ContainerStarted","Data":"7fa9ca6a85eee5063651cc30465c5ce6eb0a41fedee0aff53c70c611dde27808"} Oct 30 17:58:12.321625 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.321572 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-qnth8" podStartSLOduration=450.321554265 podStartE2EDuration="7m30.321554265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:12.320457086 +0000 UTC m=+32.825100218" watchObservedRunningTime="2026-10-30 17:58:12.321554265 +0000 UTC m=+32.826197406" Oct 30 17:58:12.321795 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.321675 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" podStartSLOduration=450.321665846 podStartE2EDuration="7m30.321665846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:12.283405474 +0000 UTC m=+32.788048614" watchObservedRunningTime="2026-10-30 17:58:12.321665846 +0000 UTC m=+32.826308984" Oct 30 17:58:12.357529 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:12.357476 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6dk47" podStartSLOduration=450.357458784 podStartE2EDuration="7m30.357458784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:12.355931041 +0000 UTC m=+32.860574180" watchObservedRunningTime="2026-10-30 17:58:12.357458784 +0000 UTC m=+32.862101922" Oct 30 17:58:13.744421 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:13.743348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:13.744421 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:13.743576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:13.744421 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:13.744095 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:13.744421 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:13.744180 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:45.744157995 +0000 UTC m=+66.248801126 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : secret "metrics-daemon-secret" not found Oct 30 17:58:13.754136 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:13.754111 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:13.854126 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:13.844880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:13.854126 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:13.852220 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:14.029587 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.029553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:14.041971 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.041899 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:14.156527 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.156134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:14.156527 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.156389 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:14.156527 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.156477 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls podName:874e6413-424d-4443-80fc-f9e4d0a63b6c nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.156457055 +0000 UTC m=+38.661100185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8g5tv" (UID: "874e6413-424d-4443-80fc-f9e4d0a63b6c") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:14.157802 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.157013 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:14.157802 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.157045 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55dcc4fcc6-4mw7c: secret "image-registry-tls" not found Oct 30 17:58:14.157802 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.157118 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls podName:056042ce-b312-4d54-9f83-15cd37aa97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.157083216 +0000 UTC m=+38.661726355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls") pod "image-registry-55dcc4fcc6-4mw7c" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce") : secret "image-registry-tls" not found Oct 30 17:58:14.157802 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.157487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:14.157802 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.157742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:14.158798 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.157888 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:14.158798 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.158066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:14.158798 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.158239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:14.158798 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.158423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:14.158798 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.158785 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:58:14.159050 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.158837 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.158822633 +0000 UTC m=+38.663465752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : secret "router-metrics-certs-default" not found Oct 30 17:58:14.159050 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.158899 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:14.159050 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.158935 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.158924737 +0000 UTC m=+38.663567862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:14.159050 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.159003 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:14.159050 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.159013 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5c5549c-bgqzt: secret "image-registry-tls" not found Oct 30 17:58:14.159050 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.159043 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls podName:93028940-79f3-45fa-be9b-6ddd8dab5633 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.159033866 +0000 UTC m=+38.663676993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls") pod "image-registry-74f5c5549c-bgqzt" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633") : secret "image-registry-tls" not found Oct 30 17:58:14.159532 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.159103 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:58:14.159532 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.159133 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls podName:d8c35677-9ec2-4c68-938f-a9cfcc872f79 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.159123603 +0000 UTC m=+38.663766724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c2d9m" (UID: "d8c35677-9ec2-4c68-938f-a9cfcc872f79") : secret "samples-operator-tls" not found Oct 30 17:58:14.159532 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.159296 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.159282485 +0000 UTC m=+38.663925604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:14.261069 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.260009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:14.261069 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.260289 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:14.261069 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:14.260347 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls podName:529241e2-eb4e-4412-ad1f-37dee4aa9b65 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.260329423 +0000 UTC m=+38.764972545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls") pod "dns-default-r94b8" (UID: "529241e2-eb4e-4412-ad1f-37dee4aa9b65") : secret "dns-default-metrics-tls" not found Oct 30 17:58:14.264367 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.263823 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" event={"ID":"0bfbeb5b-8284-4324-a8d0-95d95d23dfc6","Type":"ContainerStarted","Data":"ac86b2dd299b78cd5a71e46ad2e4e2e8a4dba91d948c9161e1181d959b48ad10"} Oct 30 17:58:14.319401 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.318779 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" podStartSLOduration=416.250668433 podStartE2EDuration="6m58.318761906s" totalImagesPullingTime="2.068093473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:14.318132998 +0000 UTC m=+34.822776137" watchObservedRunningTime="2026-10-30 17:58:14.318761906 +0000 UTC m=+34.823405036" Oct 30 17:58:14.383057 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.382983 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q74cr"] Oct 30 17:58:14.392971 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:58:14.392939 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468096db_c6fe_46bd_b256_05f2132cee4a.slice/crio-68b04e89e205b231b8c90f34b32b9d4bd6b1886358851360bf8681a376f07893 WatchSource:0}: Error finding container 68b04e89e205b231b8c90f34b32b9d4bd6b1886358851360bf8681a376f07893: Status 404 returned error can't find the container with id 68b04e89e205b231b8c90f34b32b9d4bd6b1886358851360bf8681a376f07893 Oct 30 17:58:14.427240 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:14.426228 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dt49r"] Oct 30 17:58:14.432285 ip-10-0-128-246 kubenswrapper[2576]: W1030 17:58:14.432221 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8820c7_364d_4f01_90e2_ea9979976e47.slice/crio-f8d8c65293344bf97397e4e9f6d4fd6e5b7cde9e3f4bd5fe84369abf0200ac5d WatchSource:0}: Error finding container f8d8c65293344bf97397e4e9f6d4fd6e5b7cde9e3f4bd5fe84369abf0200ac5d: Status 404 returned error can't find the container with id f8d8c65293344bf97397e4e9f6d4fd6e5b7cde9e3f4bd5fe84369abf0200ac5d Oct 30 17:58:15.270469 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:15.268546 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" event={"ID":"21caa3cf-ca4f-424a-9f40-6f91a7b176c3","Type":"ContainerStarted","Data":"a6a84143a402ae50518cfa52818f084aa6320a3cc6e5a298f23072d76477c063"} Oct 30 17:58:15.270469 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:15.269628 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:15.271651 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:15.271372 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q74cr" event={"ID":"468096db-c6fe-46bd-b256-05f2132cee4a","Type":"ContainerStarted","Data":"68b04e89e205b231b8c90f34b32b9d4bd6b1886358851360bf8681a376f07893"} Oct 30 17:58:15.271651 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:15.271395 2576 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-gxwj6 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.19:8443/readyz\": dial tcp 10.133.0.19:8443: connect: connection refused" start-of-body= Oct 30 17:58:15.271651 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:15.271437 2576 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" podUID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.19:8443/readyz\": dial tcp 10.133.0.19:8443: connect: connection refused" Oct 30 17:58:15.273356 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:15.273314 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dt49r" event={"ID":"5a8820c7-364d-4f01-90e2-ea9979976e47","Type":"ContainerStarted","Data":"f8d8c65293344bf97397e4e9f6d4fd6e5b7cde9e3f4bd5fe84369abf0200ac5d"} Oct 30 17:58:16.313505 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.313475 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" event={"ID":"b810f479-5f8c-4acc-89de-a41d288c8e2c","Type":"ContainerStarted","Data":"7c0bdfbad9a90587da858bb70fe70ae0e6fcf973890511a5dd43a249703298c5"} Oct 30 17:58:16.318935 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.318906 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zg24h" event={"ID":"476cffce-abe2-409d-9b4a-792af41d8f1c","Type":"ContainerStarted","Data":"399000c07e501b9e15a6c6646863cd8d0e4b5e5b1f5a3b8037313e120f85bd20"} Oct 30 17:58:16.321791 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.321748 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" event={"ID":"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a","Type":"ContainerStarted","Data":"72bc74cc18464467437f46d113ba54c226ba003b356fdd4fb927eaa60ca187e4"} Oct 30 17:58:16.331126 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.330530 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/0.log" Oct 30 17:58:16.331126 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.330565 2576 generic.go:356] "Generic (PLEG): container finished" podID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" containerID="a6a84143a402ae50518cfa52818f084aa6320a3cc6e5a298f23072d76477c063" exitCode=255 Oct 30 17:58:16.331126 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.330607 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" event={"ID":"21caa3cf-ca4f-424a-9f40-6f91a7b176c3","Type":"ContainerDied","Data":"a6a84143a402ae50518cfa52818f084aa6320a3cc6e5a298f23072d76477c063"} Oct 30 17:58:16.331126 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.330872 2576 scope.go:122] "RemoveContainer" containerID="a6a84143a402ae50518cfa52818f084aa6320a3cc6e5a298f23072d76477c063" Oct 30 17:58:16.348947 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.348901 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" podStartSLOduration=416.338118278 podStartE2EDuration="7m0.348886048s" totalImagesPullingTime="4.01076777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:15.310798998 +0000 UTC m=+35.815442137" watchObservedRunningTime="2026-10-30 17:58:16.348886048 +0000 UTC m=+36.853529187" Oct 30 17:58:16.395378 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.394148 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" podStartSLOduration=415.597252222 podStartE2EDuration="7m0.394129112s" totalImagesPullingTime="4.79687689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:16.350506544 +0000 UTC m=+36.855149681" watchObservedRunningTime="2026-10-30 17:58:16.394129112 +0000 UTC m=+36.898772256" Oct 30 17:58:16.441287 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.436162 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-zg24h" podStartSLOduration=416.559212014 podStartE2EDuration="7m1.436143142s" totalImagesPullingTime="4.876931128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:16.395793906 +0000 UTC m=+36.900437048" watchObservedRunningTime="2026-10-30 17:58:16.436143142 +0000 UTC m=+36.940786281" Oct 30 17:58:16.492164 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:16.492091 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" podStartSLOduration=416.594679581 podStartE2EDuration="7m1.492072295s" totalImagesPullingTime="4.897392714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:16.436818013 +0000 UTC m=+36.941461154" watchObservedRunningTime="2026-10-30 17:58:16.492072295 +0000 UTC m=+36.996715434" Oct 30 17:58:17.338171 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.338135 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s2zkr" event={"ID":"7caaae72-bd7d-44b5-902e-099da2ef429b","Type":"ContainerStarted","Data":"255d9a1af1d6eaf2404eb4269ba2d264db606854075657adf072dee3265f51c0"} Oct 30 17:58:17.340408 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.340384 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/1.log" Oct 30 17:58:17.340969 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.340946 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/0.log" Oct 30 17:58:17.341077 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.340984 2576 generic.go:356] "Generic (PLEG): container finished" podID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" containerID="d3ca2364d68fe0246dca3420e6db36a2937baca9ad9ef1d17b53de359919a379" exitCode=255 Oct 30 17:58:17.341247 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.341223 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" event={"ID":"21caa3cf-ca4f-424a-9f40-6f91a7b176c3","Type":"ContainerDied","Data":"d3ca2364d68fe0246dca3420e6db36a2937baca9ad9ef1d17b53de359919a379"} Oct 30 17:58:17.341327 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.341302 2576 scope.go:122] "RemoveContainer" containerID="a6a84143a402ae50518cfa52818f084aa6320a3cc6e5a298f23072d76477c063" Oct 30 17:58:17.341640 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.341627 2576 scope.go:122] "RemoveContainer" containerID="d3ca2364d68fe0246dca3420e6db36a2937baca9ad9ef1d17b53de359919a379" Oct 30 17:58:17.341818 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:17.341798 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gxwj6_openshift-console-operator(21caa3cf-ca4f-424a-9f40-6f91a7b176c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" podUID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" Oct 30 17:58:17.369081 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.369046 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f2zht"] Oct 30 17:58:17.384283 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.384223 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s2zkr" podStartSLOduration=1.7962257369999999 podStartE2EDuration="7.38421094s" totalImagesPullingTime="5.587985203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:17.38358521 +0000 UTC m=+37.888228350" watchObservedRunningTime="2026-10-30 17:58:17.38421094 +0000 UTC m=+37.888854112" Oct 30 17:58:17.388564 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.388542 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.391245 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.391223 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2zht"] Oct 30 17:58:17.405154 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.404789 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:58:17.405471 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.405412 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mngtd\"" Oct 30 17:58:17.408747 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.408723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.408845 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.408781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-crio-socket\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.408898 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.408847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65zz\" (UniqueName: \"kubernetes.io/projected/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-api-access-q65zz\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.409051 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.409032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-data-volume\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.409169 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.409063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.413680 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.413659 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:58:17.509983 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.509951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-data-volume\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.510155 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.509991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.510155 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.510074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.510155 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.510105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-crio-socket\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.510155 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.510138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65zz\" (UniqueName: \"kubernetes.io/projected/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-api-access-q65zz\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.510410 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:17.510188 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:17.510410 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:17.510249 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.010230269 +0000 UTC m=+38.514873409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:17.510410 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.510338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-data-volume\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.510410 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:17.510395 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:17.510539 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.510412 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-crio-socket\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:17.510539 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:17.510436 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:18.01042172 +0000 UTC m=+38.515064855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : configmap "kube-rbac-proxy" not found Oct 30 17:58:17.534075 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:17.534051 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65zz\" (UniqueName: \"kubernetes.io/projected/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-api-access-q65zz\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:18.015667 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.015584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:18.015805 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.015696 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:18.015805 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.015732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:18.015805 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.015755 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:19.01574146 +0000 UTC m=+39.520384576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : configmap "kube-rbac-proxy" not found Oct 30 17:58:18.015958 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.015830 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:18.015958 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.015885 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:19.015869946 +0000 UTC m=+39.520513065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:18.217853 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.217826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:18.217983 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.217872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:18.217983 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.217911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:18.217983 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.217939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:18.218157 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.217995 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:18.218157 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218024 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:18.218157 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218056 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls podName:874e6413-424d-4443-80fc-f9e4d0a63b6c nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.218036305 +0000 UTC m=+46.722679423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8g5tv" (UID: "874e6413-424d-4443-80fc-f9e4d0a63b6c") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:18.218157 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218063 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:18.218157 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218072 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.218061726 +0000 UTC m=+46.722704857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:18.218157 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218084 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55dcc4fcc6-4mw7c: secret "image-registry-tls" not found Oct 30 17:58:18.218157 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.218109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:18.218157 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218135 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls podName:056042ce-b312-4d54-9f83-15cd37aa97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.218117527 +0000 UTC m=+46.722760666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls") pod "image-registry-55dcc4fcc6-4mw7c" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce") : secret "image-registry-tls" not found Oct 30 17:58:18.218157 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218136 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:58:18.218541 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218166 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:18.218541 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218177 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5c5549c-bgqzt: secret "image-registry-tls" not found Oct 30 17:58:18.218541 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218187 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.218173521 +0000 UTC m=+46.722816650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : secret "router-metrics-certs-default" not found Oct 30 17:58:18.218541 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218206 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls podName:93028940-79f3-45fa-be9b-6ddd8dab5633 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.218195968 +0000 UTC m=+46.722839090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls") pod "image-registry-74f5c5549c-bgqzt" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633") : secret "image-registry-tls" not found Oct 30 17:58:18.218541 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.218223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:18.218541 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.218303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:18.218541 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218355 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:58:18.218541 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218399 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls podName:d8c35677-9ec2-4c68-938f-a9cfcc872f79 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.218386469 +0000 UTC m=+46.723029592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c2d9m" (UID: "d8c35677-9ec2-4c68-938f-a9cfcc872f79") : secret "samples-operator-tls" not found Oct 30 17:58:18.218541 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.218420 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.218406105 +0000 UTC m=+46.723049260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:18.319701 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.319674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:18.319879 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.319857 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:18.319937 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.319923 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls podName:529241e2-eb4e-4412-ad1f-37dee4aa9b65 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:26.319905109 +0000 UTC m=+46.824548230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls") pod "dns-default-r94b8" (UID: "529241e2-eb4e-4412-ad1f-37dee4aa9b65") : secret "dns-default-metrics-tls" not found Oct 30 17:58:18.348634 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.348593 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/1.log" Oct 30 17:58:18.349091 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.349057 2576 scope.go:122] "RemoveContainer" containerID="d3ca2364d68fe0246dca3420e6db36a2937baca9ad9ef1d17b53de359919a379" Oct 30 17:58:18.349362 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:18.349329 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gxwj6_openshift-console-operator(21caa3cf-ca4f-424a-9f40-6f91a7b176c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" podUID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" Oct 30 17:58:18.350900 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.350872 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" event={"ID":"df983796-b36d-40f0-ae14-93ba96e518e3","Type":"ContainerStarted","Data":"efab25b67c3372d60a6344e40c119df9a595c6e401ff13a0e6a92dcca30afcc3"} Oct 30 17:58:18.353067 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.353030 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q74cr" event={"ID":"468096db-c6fe-46bd-b256-05f2132cee4a","Type":"ContainerStarted","Data":"14dc8544bd0f9faa4e1ebd7aa9b751d67bbe088660dc798c3dcacccb4662951f"} Oct 30 17:58:18.434972 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.434928 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q74cr" podStartSLOduration=33.744741485 podStartE2EDuration="37.434911152s" totalImagesPullingTime="3.690169667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:18.434487384 +0000 UTC m=+38.939130521" watchObservedRunningTime="2026-10-30 17:58:18.434911152 +0000 UTC m=+38.939554294" Oct 30 17:58:18.471298 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:18.471224 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" podStartSLOduration=413.98327848 podStartE2EDuration="7m0.471207696s" totalImagesPullingTime="6.487929216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:18.470442649 +0000 UTC m=+38.975085789" watchObservedRunningTime="2026-10-30 17:58:18.471207696 +0000 UTC m=+38.975850836" Oct 30 17:58:19.026453 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:19.026412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:19.026644 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:19.026543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:19.026644 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:19.026553 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 17:58:19.026644 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:19.026636 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:21.026612689 +0000 UTC m=+41.531255829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : configmap "kube-rbac-proxy" not found Oct 30 17:58:19.026818 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:19.026677 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:19.026818 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:19.026731 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:21.026716787 +0000 UTC m=+41.531359921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:19.357202 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:19.357168 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:20.745868 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:20.745833 2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:20.746368 ip-10-0-128-246 kubenswrapper[2576]: I1030 17:58:20.746153 2576 scope.go:122] "RemoveContainer" containerID="d3ca2364d68fe0246dca3420e6db36a2937baca9ad9ef1d17b53de359919a379" Oct 30 17:58:20.746368 ip-10-0-128-246 kubenswrapper[2576]: E1030 17:58:20.746332 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gxwj6_openshift-console-operator(21caa3cf-ca4f-424a-9f40-6f91a7b176c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" podUID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" Oct 30 17:58:20.855939 ip-10-0-128-246 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:58:20.880667 ip-10-0-128-246 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:58:20.880841 ip-10-0-128-246 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:58:20.880890 ip-10-0-128-246 systemd[1]: kubelet.service: Consumed 2.879s CPU time. Oct 30 17:58:20.895013 ip-10-0-128-246 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:58:20.980458 ip-10-0-128-246 kubenswrapper[4862]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:58:20.980458 ip-10-0-128-246 kubenswrapper[4862]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:58:20.980458 ip-10-0-128-246 kubenswrapper[4862]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:58:20.980458 ip-10-0-128-246 kubenswrapper[4862]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:58:20.982518 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982443 4862 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:20.982518 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982517 4862 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:20.982518 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982521 4862 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982524 4862 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982527 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982530 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982533 4862 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982536 4862 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982538 4862 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982541 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982543 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982546 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982548 4862 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982551 4862 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982553 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982556 4862 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982558 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982561 4862 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982563 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982570 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982573 4862 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982575 4862 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:20.982626 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982578 4862 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982580 4862 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982583 4862 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982585 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982588 4862 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982590 4862 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982593 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982595 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982598 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982600 4862 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982603 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982606 4862 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982609 4862 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982628 4862 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982631 4862 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982634 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982637 4862 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982640 4862 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982643 4862 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:20.983140 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982646 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982648 4862 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982651 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982653 4862 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982656 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982658 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982661 4862 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982663 4862 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982666 4862 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982669 4862 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982671 4862 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982673 4862 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982676 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982679 4862 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982681 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982684 4862 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982686 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982689 4862 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982691 4862 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982694 4862 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:20.983844 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982696 4862 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982699 4862 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982702 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982705 4862 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982707 4862 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982710 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982712 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982715 4862 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982718 4862 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982720 4862 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982723 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982726 4862 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982728 4862 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982731 4862 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982733 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982736 4862 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982739 4862 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982741 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982744 4862 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982746 4862 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:20.984342 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982749 4862 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982751 4862 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982754 4862 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982756 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982759 4862 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982761 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982764 4862 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982767 4862 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982770 4862 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982774 4862 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982778 4862 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982782 4862 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982785 4862 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982787 4862 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982790 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982794 4862 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982798 4862 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982801 4862 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982804 4862 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:20.984852 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982807 4862 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982809 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982812 4862 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982815 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982817 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.982820 4862 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983587 4862 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983594 4862 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983597 4862 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983601 4862 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983604 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983606 4862 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983609 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983611 4862 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983614 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983616 4862 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983619 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983621 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983624 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983627 4862 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:20.985343 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983629 4862 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983632 4862 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983634 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983637 4862 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983639 4862 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983641 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983644 4862 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983646 4862 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983650 4862 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983652 4862 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983655 4862 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983658 4862 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983660 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983663 4862 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983665 4862 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983668 4862 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983670 4862 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983673 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983676 4862 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:20.985831 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983679 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983682 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983685 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983687 4862 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983690 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983692 4862 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983694 4862 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983697 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983699 4862 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983702 4862 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983704 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983707 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983709 4862 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983712 4862 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983714 4862 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983716 4862 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983719 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983722 4862 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983724 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:20.986322 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983726 4862 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983729 4862 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983731 4862 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983734 4862 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983737 4862 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983739 4862 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983742 4862 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983744 4862 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983747 4862 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983749 4862 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983752 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983754 4862 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983757 4862 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983760 4862 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983763 4862 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983765 4862 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983769 4862 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983773 4862 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983776 4862 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:20.986841 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983778 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983781 4862 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983783 4862 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983786 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983789 4862 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983791 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983794 4862 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983797 4862 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983799 4862 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983802 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983804 4862 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983807 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983809 4862 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983812 4862 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983814 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983817 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983822 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983825 4862 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983827 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:20.987340 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983830 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983832 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983835 4862 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983837 4862 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983840 4862 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983843 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983845 4862 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983848 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983851 4862 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983854 4862 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983857 4862 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983861 4862 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983866 4862 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983869 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983871 4862 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.983874 4862 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984365 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984371 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984373 4862 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984376 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:20.987802 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984378 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984383 4862 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984387 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984391 4862 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984394 4862 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984396 4862 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984399 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984402 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984404 4862 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984407 4862 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984410 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984413 4862 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984415 4862 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984418 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984420 4862 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984423 4862 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984425 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984427 4862 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984430 4862 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:20.988350 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984432 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984435 4862 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984438 4862 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984440 4862 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984443 4862 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984445 4862 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984447 4862 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984450 4862 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984452 4862 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984455 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984457 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984459 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984462 4862 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984464 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984466 4862 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984469 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984471 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984474 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984476 4862 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984481 4862 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:20.988821 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984484 4862 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984487 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984489 4862 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984492 4862 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984495 4862 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984497 4862 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984500 4862 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984502 4862 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984505 4862 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984507 4862 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984510 4862 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984512 4862 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984515 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984517 4862 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984521 4862 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984523 4862 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984526 4862 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984528 4862 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984531 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984534 4862 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:20.989337 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984537 4862 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984539 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984541 4862 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984544 4862 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984546 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984548 4862 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984551 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984553 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984556 4862 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984558 4862 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984560 4862 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984564 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984567 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984569 4862 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984572 4862 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984574 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984577 4862 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984580 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984582 4862 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:20.989986 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984585 4862 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984587 4862 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984590 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984592 4862 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984595 4862 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984597 4862 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984600 4862 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984602 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984605 4862 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984608 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984610 4862 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984613 4862 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984615 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984618 4862 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984620 4862 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984622 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984625 4862 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984627 4862 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984630 4862 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984632 4862 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:20.990753 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984635 4862 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984638 4862 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984642 4862 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.984646 4862 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984740 4862 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984752 4862 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984758 4862 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984763 4862 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984767 4862 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984770 4862 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984774 4862 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984778 4862 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984781 4862 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984784 4862 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984788 4862 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984791 4862 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984794 4862 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984797 4862 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984800 4862 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984803 4862 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984806 4862 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984809 4862 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984813 4862 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:58:20.991253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984816 4862 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984819 4862 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984822 4862 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984825 4862 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984829 4862 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984832 4862 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984835 4862 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984838 4862 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984841 4862 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984844 4862 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984847 4862 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984853 4862 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984855 4862 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984861 4862 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984864 4862 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984867 4862 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984870 4862 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984873 4862 flags.go:64] FLAG: --enable-server="true" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984876 4862 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984879 4862 flags.go:64] FLAG: --event-burst="100" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984882 4862 flags.go:64] FLAG: --event-qps="50" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984885 4862 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984888 4862 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984891 4862 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:58:20.991843 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984895 4862 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984898 4862 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984901 4862 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984904 4862 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984907 4862 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984909 4862 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984912 4862 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984916 4862 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984919 4862 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984922 4862 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984924 4862 flags.go:64] FLAG: --feature-gates="" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984929 4862 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984932 4862 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984935 4862 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984938 4862 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984941 4862 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984944 4862 flags.go:64] FLAG: --help="false" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984947 4862 flags.go:64] FLAG: --hostname-override="ip-10-0-128-246.ec2.internal" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984950 4862 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984954 4862 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984957 4862 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984961 4862 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984964 4862 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984967 4862 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:58:20.992481 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984970 4862 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984973 4862 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984976 4862 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984979 4862 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984982 4862 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984984 4862 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984987 4862 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984990 4862 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984993 4862 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984995 4862 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.984998 4862 flags.go:64] FLAG: --lock-file="" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985001 4862 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985004 4862 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985007 4862 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985018 4862 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985021 4862 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985025 4862 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985028 4862 flags.go:64] FLAG: --logging-format="text" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985030 4862 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985034 4862 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985044 4862 flags.go:64] FLAG: --manifest-url="" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985047 4862 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985051 4862 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985054 4862 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985058 4862 flags.go:64] FLAG: --max-pods="110" Oct 30 17:58:20.993088 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985061 4862 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985064 4862 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985068 4862 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985071 4862 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985075 4862 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985078 4862 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985081 4862 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985088 4862 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985090 4862 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985093 4862 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985096 4862 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985099 4862 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985102 4862 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985105 4862 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985107 4862 flags.go:64] FLAG: --port="10250" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985110 4862 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985113 4862 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05548694925729995" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985116 4862 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985119 4862 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985121 4862 flags.go:64] FLAG: --register-node="true" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985124 4862 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985128 4862 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985131 4862 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985134 4862 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985137 4862 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985140 4862 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:58:20.993703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985143 4862 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985146 4862 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985149 4862 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985152 4862 flags.go:64] FLAG: --runonce="false" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985155 4862 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985158 4862 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985161 4862 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985163 4862 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985167 4862 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985170 4862 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985174 4862 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985177 4862 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985180 4862 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985183 4862 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985185 4862 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985188 4862 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985191 4862 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985194 4862 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985197 4862 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985202 4862 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985204 4862 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985207 4862 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985211 4862 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985214 4862 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985217 4862 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:58:20.994374 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985220 4862 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985223 4862 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985225 4862 flags.go:64] FLAG: --v="2" Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985229 4862 flags.go:64] FLAG: --version="false" Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985233 4862 flags.go:64] FLAG: --vmodule="" Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985237 4862 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985241 4862 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985395 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985400 4862 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985403 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985406 4862 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985408 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985411 4862 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985414 4862 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985417 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985421 4862 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985423 4862 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985427 4862 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985430 4862 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985432 4862 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985435 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:20.994964 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985438 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985440 4862 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985443 4862 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985445 4862 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985448 4862 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985450 4862 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985453 4862 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985455 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985457 4862 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985460 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985462 4862 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985465 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985468 4862 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985470 4862 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985473 4862 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985475 4862 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985477 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985480 4862 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985483 4862 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:20.995534 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985485 4862 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985488 4862 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985491 4862 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985495 4862 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985499 4862 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985503 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985506 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985510 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985513 4862 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985517 4862 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985519 4862 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985522 4862 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985524 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985527 4862 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985529 4862 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985532 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985534 4862 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985537 4862 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985539 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:20.996157 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985542 4862 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985544 4862 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985547 4862 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985549 4862 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985552 4862 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985555 4862 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985557 4862 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985560 4862 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985562 4862 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985564 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985567 4862 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985569 4862 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985572 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985575 4862 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985578 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985580 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985582 4862 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985585 4862 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985587 4862 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985590 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:20.996790 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985596 4862 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985599 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985602 4862 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985605 4862 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985607 4862 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985610 4862 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985613 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985615 4862 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985618 4862 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985620 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985623 4862 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985625 4862 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985628 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985630 4862 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985633 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985635 4862 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985639 4862 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985642 4862 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985646 4862 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985649 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:20.997297 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985651 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985654 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985656 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985659 4862 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985661 4862 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985664 4862 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985667 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985669 4862 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985672 4862 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985674 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985677 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985679 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985683 4862 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985685 4862 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.985712 4862 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:58:20.997880 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985745 4862 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985749 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985753 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985755 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985758 4862 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985761 4862 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985763 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985766 4862 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985769 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985771 4862 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985774 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985776 4862 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985778 4862 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985781 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985783 4862 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985786 4862 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985788 4862 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985791 4862 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985793 4862 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985796 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:20.998315 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985798 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985801 4862 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985803 4862 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985806 4862 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985809 4862 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985811 4862 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985814 4862 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985816 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985819 4862 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985822 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985825 4862 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985828 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985831 4862 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985834 4862 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985836 4862 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985839 4862 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985842 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985844 4862 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985847 4862 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:20.998869 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985849 4862 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985852 4862 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985855 4862 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985859 4862 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985862 4862 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985865 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985869 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985871 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985874 4862 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985878 4862 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985880 4862 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985883 4862 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985885 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985888 4862 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985890 4862 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985893 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985895 4862 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985898 4862 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985901 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:20.999362 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985903 4862 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985906 4862 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985908 4862 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985912 4862 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985915 4862 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985917 4862 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985920 4862 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985923 4862 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985925 4862 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985927 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985930 4862 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985932 4862 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985934 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985937 4862 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985939 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985942 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985944 4862 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985947 4862 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985949 4862 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985952 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:20.999825 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985954 4862 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985956 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985959 4862 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985961 4862 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985963 4862 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985966 4862 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985968 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985971 4862 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985973 4862 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985975 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985978 4862 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985980 4862 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985983 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985986 4862 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985988 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985991 4862 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985996 4862 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.985999 4862 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986002 4862 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986006 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:21.000435 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986009 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986011 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986014 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986016 4862 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986019 4862 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986021 4862 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986024 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.986026 4862 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.990458 4862 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.990477 4862 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.990500 4862 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.990507 4862 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990892 4862 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990906 4862 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990910 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990913 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990916 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990919 4862 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990921 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:21.000940 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990924 4862 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990926 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990930 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990934 4862 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990936 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990939 4862 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990941 4862 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990944 4862 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990946 4862 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990949 4862 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990953 4862 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990957 4862 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990959 4862 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990962 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990965 4862 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990968 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990970 4862 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990973 4862 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990977 4862 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990980 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:21.001628 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990982 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990986 4862 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990989 4862 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990992 4862 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990994 4862 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.990997 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991000 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991002 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991005 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991007 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991010 4862 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991012 4862 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991015 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991017 4862 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991020 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991023 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991025 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991028 4862 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991031 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991033 4862 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:21.002153 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991036 4862 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991039 4862 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991041 4862 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991045 4862 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991047 4862 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991050 4862 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991053 4862 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991055 4862 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991057 4862 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991060 4862 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991063 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991065 4862 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991068 4862 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991071 4862 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991074 4862 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991077 4862 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991081 4862 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991085 4862 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991089 4862 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:21.002651 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991092 4862 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991094 4862 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991098 4862 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991102 4862 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991105 4862 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991107 4862 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991110 4862 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991113 4862 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991116 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991118 4862 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991121 4862 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991123 4862 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991126 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991135 4862 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991138 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991140 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991143 4862 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991145 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991148 4862 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:21.003098 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991150 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991153 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991155 4862 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991158 4862 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991161 4862 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991163 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991166 4862 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991168 4862 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991171 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991173 4862 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991176 4862 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991179 4862 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991182 4862 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991184 4862 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991187 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991189 4862 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991192 4862 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991194 4862 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991196 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:21.003578 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991199 4862 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991202 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.991232 4862 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991404 4862 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991410 4862 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991412 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991415 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991418 4862 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991426 4862 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991429 4862 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991432 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991434 4862 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991437 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991439 4862 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991442 4862 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991444 4862 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:21.004026 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991447 4862 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991449 4862 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991452 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991455 4862 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991457 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991460 4862 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991462 4862 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991465 4862 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991467 4862 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991470 4862 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991472 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991475 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991477 4862 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991479 4862 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991482 4862 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991484 4862 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991487 4862 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991489 4862 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991491 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991494 4862 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:21.004419 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991496 4862 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991499 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991501 4862 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991504 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991506 4862 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991509 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991512 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991514 4862 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991517 4862 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991519 4862 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991521 4862 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991524 4862 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991526 4862 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991528 4862 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991531 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991534 4862 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991536 4862 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991541 4862 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991543 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:21.004900 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991547 4862 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991550 4862 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991553 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991556 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991558 4862 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991561 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991563 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991566 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991568 4862 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991570 4862 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991573 4862 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991575 4862 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991578 4862 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991581 4862 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991585 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991588 4862 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991590 4862 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991592 4862 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991595 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:21.005437 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991598 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991600 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991603 4862 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991605 4862 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991608 4862 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991611 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991613 4862 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991615 4862 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991618 4862 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991621 4862 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991623 4862 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991627 4862 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991629 4862 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991631 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991634 4862 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991636 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991639 4862 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991641 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991643 4862 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:21.005917 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991646 4862 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991649 4862 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991651 4862 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991654 4862 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991656 4862 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991658 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991661 4862 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991664 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991666 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991668 4862 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991671 4862 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991673 4862 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991676 4862 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991678 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991681 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991684 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:21.006380 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.991707 4862 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991736 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991740 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991743 4862 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991747 4862 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991749 4862 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991752 4862 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991755 4862 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991760 4862 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991763 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991766 4862 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991769 4862 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991772 4862 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991775 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991777 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991779 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991782 4862 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991784 4862 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991787 4862 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:58:21.006980 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991790 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991792 4862 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991795 4862 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991797 4862 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991799 4862 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991802 4862 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991804 4862 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991807 4862 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991809 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991812 4862 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991814 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991816 4862 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991819 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991822 4862 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991824 4862 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991827 4862 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991830 4862 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991834 4862 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991836 4862 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991839 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:58:21.007771 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991841 4862 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991845 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991848 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991850 4862 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991853 4862 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991855 4862 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991858 4862 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991861 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991863 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991866 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991869 4862 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991871 4862 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991874 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991876 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991879 4862 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991882 4862 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991884 4862 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991887 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991889 4862 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:58:21.008524 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991892 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991894 4862 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991897 4862 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991899 4862 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991902 4862 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991904 4862 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991907 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991910 4862 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991912 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991915 4862 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991919 4862 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991921 4862 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991924 4862 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991926 4862 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991930 4862 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991933 4862 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991935 4862 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991937 4862 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991940 4862 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991942 4862 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:58:21.009045 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991945 4862 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991947 4862 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991950 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991952 4862 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991955 4862 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991957 4862 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991960 4862 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991962 4862 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991965 4862 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991967 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991970 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991972 4862 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991975 4862 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991977 4862 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991980 4862 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991982 4862 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991985 4862 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991987 4862 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991990 4862 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:58:21.009598 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991993 4862 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991995 4862 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.991997 4862 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.992001 4862 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.992006 4862 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.992009 4862 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.992012 4862 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.992016 4862 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.992019 4862 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:20.992022 4862 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.992100 4862 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.993043 4862 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.993101 4862 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.993301 4862 server.go:1014] "Starting client certificate rotation" Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.993443 4862 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.993498 4862 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:52:40 +0000 UTC" deadline="2028-05-12 12:19:53.128318241 +0000 UTC" Oct 30 17:58:21.010091 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.993532 4862 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13434h21m32.134788318s" Oct 30 17:58:21.010600 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.993734 4862 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:58:21.010600 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.994846 4862 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:58:21.010600 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.996059 4862 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:58:21.010600 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.997716 4862 log.go:25] "Validated CRI v1 image API" Oct 30 17:58:21.010600 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:20.998489 4862 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:58:21.010600 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.000210 4862 fs.go:135] Filesystem UUIDs: map[4f255b88-c943-4a4e-964a-1b5d252f5ff3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e80626bd-5894-4990-beb4-8b6fb3ae231f:/dev/nvme0n1p3] Oct 30 17:58:21.010777 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.000228 4862 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0311e96784e88a5761844fee94b5e902487d37e562f249063e7a567a5c866402/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0311e96784e88a5761844fee94b5e902487d37e562f249063e7a567a5c866402/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0654be5cdc8e86bc08fae1da65afa1855395fa4993dee17b4990e536079479a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0654be5cdc8e86bc08fae1da65afa1855395fa4993dee17b4990e536079479a6/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07ea06b86bef3a20ce9115a13047804abe6ed2b3f536afcb024421693455c5aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07ea06b86bef3a20ce9115a13047804abe6ed2b3f536afcb024421693455c5aa/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e0b6fd527e592f71ba6e70227f28f619dcdcb162b4ff7b24db493bec7884b17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e0b6fd527e592f71ba6e70227f28f619dcdcb162b4ff7b24db493bec7884b17/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2558988e914b7ff0ecf6b2ad7c05113d298e2106d7f045367df2267c79f09499/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2558988e914b7ff0ecf6b2ad7c05113d298e2106d7f045367df2267c79f09499/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4606c18ee91f4eddbb933f9a86249a4e84f54314c3523d8f76e1db8321eb2f5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4606c18ee91f4eddbb933f9a86249a4e84f54314c3523d8f76e1db8321eb2f5a/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65a1ff50dc1d7c32bf2b223fd3cf3bb7daa2be665fdec8fd3cc474af1a0fc815/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65a1ff50dc1d7c32bf2b223fd3cf3bb7daa2be665fdec8fd3cc474af1a0fc815/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68b04e89e205b231b8c90f34b32b9d4bd6b1886358851360bf8681a376f07893/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68b04e89e205b231b8c90f34b32b9d4bd6b1886358851360bf8681a376f07893/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e212f4ef613c4cf68f2ead0714d8b305f4eaa39361270349e41c22b0cb99416/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e212f4ef613c4cf68f2ead0714d8b305f4eaa39361270349e41c22b0cb99416/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72430c9337e74812ba6b2ad90d827aa84f53a792941dbda4e33fd28e92f264fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72430c9337e74812ba6b2ad90d827aa84f53a792941dbda4e33fd28e92f264fa/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f4e39ad52d0ec41bac2e39cda30df3b4208773218386c5ffd2f44f991528f5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f4e39ad52d0ec41bac2e39cda30df3b4208773218386c5ffd2f44f991528f5a/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fa9ca6a85eee5063651cc30465c5ce6eb0a41fedee0aff53c70c611dde27808/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fa9ca6a85eee5063651cc30465c5ce6eb0a41fedee0aff53c70c611dde27808/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95855e4c59040c66b37ef72a54cb9de5487c85997d619f22c84accec198ab5be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95855e4c59040c66b37ef72a54cb9de5487c85997d619f22c84accec198ab5be/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96dd8351e8c09ab7b93fdf0d7494b9a33c1bd4c0b95c281ef9a60055a8c7341f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96dd8351e8c09ab7b93fdf0d7494b9a33c1bd4c0b95c281ef9a60055a8c7341f/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a78c392cc5757ee37c6a65b839638dc72ac734944b3661dc542cb126d0044288/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a78c392cc5757ee37c6a65b839638dc72ac734944b3661dc542cb126d0044288/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abe3cbfd004f98d7f1a99e6dfda84ed966a56a5b4b4252334ff3f9b8fd6af697/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abe3cbfd004f98d7f1a99e6dfda84ed966a56a5b4b4252334ff3f9b8fd6af697/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b518a841587144a85ff55bd0dfc07e01e250dec1d1cc732ba4299f848eaf0ea6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b518a841587144a85ff55bd0dfc07e01e250dec1d1cc732ba4299f848eaf0ea6/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba92e9d064ca58f324001642abb2d2f5ae8fdb9b3acb7feb20f0f7a85ff9c522/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba92e9d064ca58f324001642abb2d2f5ae8fdb9b3acb7feb20f0f7a85ff9c522/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e697292bee026df8571cc20abfa9fe1a8186b18c54128a40553fcbbc3fac837d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e697292bee026df8571cc20abfa9fe1a8186b18c54128a40553fcbbc3fac837d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8d8c65293344bf97397e4e9f6d4fd6e5b7cde9e3f4bd5fe84369abf0200ac5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8d8c65293344bf97397e4e9f6d4fd6e5b7cde9e3f4bd5fe84369abf0200ac5d/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa3a9cce436bca035669a359cef965751407914479af843ddae0087d5a983ecd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa3a9cce436bca035669a359cef965751407914479af843ddae0087d5a983ecd/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc6bbf56c484f44ef737360c94843ce9eb85de60ec726eeb3d15cfa53a262413/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc6bbf56c484f44ef737360c94843ce9eb85de60ec726eeb3d15cfa53a262413/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~projected/kube-api-access-bx4ss:{mountpoint:/var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~projected/kube-api-access-bx4ss major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bfbeb5b-8284-4324-a8d0-95d95d23dfc6/volumes/kubernetes.io~projected/kube-api-access-7hrbz:{mountpoint:/var/lib/kubelet/pods/0bfbeb5b-8284-4324-a8d0-95d95d23dfc6/volumes/kubernetes.io~projected/kube-api-access-7hrbz major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bc68151-0620-48d2-a803-e7cb588c5b76/volumes/kubernetes.io~projected/kube-api-access-4zm6w:{mountpoint:/var/lib/kubelet/pods/1bc68151-0620-48d2-a803-e7cb588c5b76/volumes/kubernetes.io~projected/kube-api-access-4zm6w major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21caa3cf-ca4f-424a-9f40-6f91a7b176c3/volumes/kubernetes.io~projected/kube-api-access-cz7ww:{mountpoint:/var/lib/kubelet/pods/21caa3cf-ca4f-424a-9f40-6f91a7b176c3/volumes/kubernetes.io~projected/kube-api-access-cz7ww major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21caa3cf-ca4f-424a-9f40-6f91a7b176c3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/21caa3cf-ca4f-424a-9f40-6f91a7b176c3/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/468096db-c6fe-46bd-b256-05f2132cee4a/volumes/kubernetes.io~projected/kube-api-access-flk52:{mountpoint:/var/lib/kubelet/pods/468096db-c6fe-46bd-b256-05f2132cee4a/volumes/kubernetes.io~projected/kube-api-access-flk52 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/476cffce-abe2-409d-9b4a-792af41d8f1c/volumes/kubernetes.io~projected/kube-api-access-4lhrg:{mountpoint:/var/lib/kubelet/pods/476cffce-abe2-409d-9b4a-792af41d8f1c/volumes/kubernetes.io~projected/kube-api-access-4lhrg major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/476cffce-abe2-409d-9b4a-792af41d8f1c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/476cffce-abe2-409d-9b4a-792af41d8f1c/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f34929e-0b21-4743-af11-1360d39f162b/volumes/kubernetes.io~projected/kube-api-access-p7fzn:{mountpoint:/var/lib/kubelet/pods/4f34929e-0b21-4743-af11-1360d39f162b/volumes/kubernetes.io~projected/kube-api-access-p7fzn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50098568-1891-40d1-886a-9065582c2467/volumes/kubernetes.io~projected/kube-api-access-hlzws:{mountpoint:/var/lib/kubelet/pods/50098568-1891-40d1-886a-9065582c2467/volumes/kubernetes.io~projected/kube-api-access-hlzws major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/529241e2-eb4e-4412-ad1f-37dee4aa9b65/volumes/kubernetes.io~projected/kube-api-access-bnl8d:{mountpoint:/var/lib/kubelet/pods/529241e2-eb4e-4412-ad1f-37dee4aa9b65/volumes/kubernetes.io~projected/kube-api-access-bnl8d major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56c4dbcb-db0a-4e7d-a633-3ff9304037d4/volumes/kubernetes.io~projected/kube-api-access-dgvqb:{mountpoint:/var/lib/kubelet/pods/56c4dbcb-db0a-4e7d-a633-3ff9304037d4/volumes/kubernetes.io~projected/kube-api-access-dgvqb major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a/volumes/kubernetes.io~projected/kube-api-access-5g8s9:{mountpoint:/var/lib/kubelet/pods/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a/volumes/kubernetes.io~projected/kube-api-access-5g8s9 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a311faf-5bcd-47ab-b102-62aa0abf168c/volumes/kubernetes.io~projected/kube-api-access-mp5lz:{mountpoint:/var/lib/kubelet/pods/5a311faf-5bcd-47ab-b102-62aa0abf168c/volumes/kubernetes.io~projected/kube-api-access-mp5lz major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8820c7-364d-4f01-90e2-ea9979976e47/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5a8820c7-364d-4f01-90e2-ea9979976e47/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b5e199e-2896-450f-9e73-838fb7e12e48/volumes/kubernetes.io~projected/kube-api-access-k6zgv:{mountpoint:/var/lib/kubelet/pods/5b5e199e-2896-450f-9e73-838fb7e12e48/volumes/kubernetes.io~projected/kube-api-access-k6zgv major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b5e199e-2896-450f-9e73-838fb7e12e48/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/5b5e199e-2896-450f-9e73-838fb7e12e48/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b5e199e-2896-450f-9e73-838fb7e12e48/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/5b5e199e-2896-450f-9e73-838fb7e12e48/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66e91a23-8415-48a9-924c-64f752b59272/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/66e91a23-8415-48a9-924c-64f752b59272/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66e91a23-8415-48a9-924c-64f752b59272/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/66e91a23-8415-48a9-924c-64f752b59272/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66e91a23-8415-48a9-924c-64f752b59272/volumes/kubernetes.io~projected/kube-api-access-59rfw:{mountpoint:/var/lib/kubelet/pods/66e91a23-8415-48a9-924c-64f752b59272/volumes/kubernetes.io~projected/kube-api-access-59rfw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7/volumes/kubernetes.io~projected/kube-api-access-xlqs4:{mountpoint:/var/lib/kubelet/pods/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7/volumes/kubernetes.io~projected/kube-api-access-xlqs4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7caaae72-bd7d-44b5-902e-099da2ef429b/volumes/kubernetes.io~projected/kube-api-access-fwrhb:{mountpoint:/var/lib/kubelet/pods/7caaae72-bd7d-44b5-902e-099da2ef429b/volumes/kubernetes.io~projected/kube-api-access-fwrhb major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7caaae72-bd7d-44b5-902e-099da2ef429b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7caaae72-bd7d-44b5-902e-099da2ef429b/volumes/kubernetes.io~secret/cert major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f619d36-116d-4434-bf1e-1aecb4d7fefe/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7f619d36-116d-4434-bf1e-1aecb4d7fefe/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/874e6413-424d-4443-80fc-f9e4d0a63b6c/volumes/kubernetes.io~projected/kube-api-access-9mklf:{mountpoint:/var/lib/kubelet/pods/874e6413-424d-4443-80fc-f9e4d0a63b6c/volumes/kubernetes.io~projected/kube-api-access-9mklf major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~projected/kube-api-access-cl9jp:{mountpoint:/var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~projected/kube-api-access-cl9jp major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40/volumes/kubernetes.io~projected/kube-api-access-cg7gv:{mountpoint:/var/lib/kubelet/pods/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40/volumes/kubernetes.io~projected/kube-api-access-cg7gv major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab36263f-c154-4f9f-9a8f-10fc38405e52/volumes/kubernetes.io~projected/kube-api-access-n7qcx:{mountpoint:/var/lib/kubelet/pods/ab36263f-c154-4f9f-9a8f-10fc38405e52/volumes/kubernetes.io~projected/kube-api-access-n7qcx major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b810f479-5f8c-4acc-89de-a41d288c8e2c/volumes/kubernetes.io~projected/kube-api-access-hrnqk:{mountpoint:/var/lib/kubelet/pods/b810f479-5f8c-4acc-89de-a41d288c8e2c/volumes/kubernetes.io~projected/kube-api-access-hrnqk major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b810f479-5f8c-4acc-89de-a41d288c8e2c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b810f479-5f8c-4acc-89de-a41d288c8e2c/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c598714d-0188-42bd-80e6-d9873dc2ae20/volumes/kubernetes.io~projected/kube-api-access-6dwv6:{mountpoint:/var/lib/kubelet/pods/c598714d-0188-42bd-80e6-d9873dc2ae20/volumes/kubernetes.io~projected/kube-api-access-6dwv6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd/volumes/kubernetes.io~projected/kube-api-access-q65zz:{mountpoint:/var/lib/kubelet/pods/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd/volumes/kubernetes.io~projected/kube-api-access-q65zz major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79402ab-1f1f-458b-b7c8-3c8209e69efb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d79402ab-1f1f-458b-b7c8-3c8209e69efb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79402ab-1f1f-458b-b7c8-3c8209e69efb/volumes/kubernetes.io~projected/kube-api-access-tlxpz:{mountpoint:/var/lib/kubelet/pods/d79402ab-1f1f-458b-b7c8-3c8209e69efb/volumes/kubernetes.io~projected/kube-api-access-tlxpz major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d79402ab-1f1f-458b-b7c8-3c8209e69efb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d79402ab-1f1f-458b-b7c8-3c8209e69efb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c35677-9ec2-4c68-938f-a9cfcc872f79/volumes/kubernetes.io~projected/kube-api-access-pvzmq:{mountpoint:/var/lib/kubelet/pods/d8c35677-9ec2-4c68-938f-a9cfcc872f79/volumes/kubernetes.io~projected/kube-api-access-pvzmq major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df983796-b36d-40f0-ae14-93ba96e518e3/volumes/kubernetes.io~projected/kube-api-access-6fc78:{mountpoint:/var/lib/kubelet/pods/df983796-b36d-40f0-ae14-93ba96e518e3/volumes/kubernetes.io~projected/kube-api-access-6fc78 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f11682e3-9202-425f-af04-da7b7c2351be/volumes/kubernetes.io~projected/kube-api-access-tj8ms:{mountpoint:/var/lib/kubelet/pods/f11682e3-9202-425f-af04-da7b7c2351be/volumes/kubernetes.io~projected/kube-api-access-tj8ms major:0 minor:213 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8d5abf73a9a5db3a16988061acd70f2a96b65639621fbeb2763108e73600fbf9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3d9fa4dde9f3847ef7771880eb2d1d4a15c952117c72e76b5886fca47c586fa6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/be5553d117470a8199a0be0466a65cd5a61b92c6c052afb51e55bc0df6efea73/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/016ba0a4041f2a05645db88548efaa29ec082e42304f32e30631af0f270bf817/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fb07027afdbd9bb7472e48b7f4c03213779bf84c7df5f7ede59a6ba5f0904ebb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/67d9646b4f223720535ad636cd632e5aaf0c767cb1ccf76858ad4a1a6e3dd435/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ea1b57d462b3987f435232cd858f7b44ddbc275be9b25a88983a52eb23bb1781/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/bd86877a29d00a34dd288ef6a0a6b8d0f07e20f848f5c584a7108c1c8dbda23e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a2a2c0ba76e596f7b069ecac2631f81e3f481059ff5db59ae2c58b2eb85570c4/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ae34158de77a3aebaa52e0cbfe4f4a1c807b540d60ad895a77c41991e6bb10e1/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/27259416e1c9ae5d824264445a526fc525e64a201a1728735197f2988e0bce61/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/cd0bb29990e5d315ca94390498518ac06769d1d6f03752d5bac0df92ed7bd6fa/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0b17887cdfb238bc3293fae486e04dee1fbc34899f13bd2a9f33db4d52eedd1c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/8ee1e7290728eb34c8cfb864ae3812689562c917859d805df77705b524419090/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d39b9a3c1c059db35189666a5e0fed02019c7c043da0c6d2905ae3be1ffa0071/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0f61f8d6d9f98e9ec7d391f242825b2060e4fcd941747b4654225077508350e6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/cd9af57be9ad22449ddb97d0e6c194004d89d3f7c2cf23d55720b19d07026540/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/9e19e1a7954ce6f28f6313c2197d700032c82f64aeeb513be3626899f840cff6/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ffdf6007859a22e7f80fdb0c887f14e0833169390bc825f32232900bcef0329d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/93a6f1123422ace17eaa6628c7e3e81bf6110e4b889cd081ba171d26b801347c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2804fd01d070f19c5f431d68488b3bc9eae2363825e86e90f090edcefbfa5f85/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5aa227da64adfb54043d2481e4ea532ab4b8be31517dc450c0d983c82cb14948/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/fa0a87520cffeb9f9771a7ac0befbefb0157d17225ca6ebc24c02e76010a01d5/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f527f29477396f78e067a99d35216ad3d758017ff16f25e1040d2b5c6bf0821a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/a3cc76b133f0ae2e705a47052b60c395aa19c6f96d43dd78fd9403d68951ec29/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f68a26d968a544159c2e5f3d4b2891a6921d17e35087ddfb0f6bb88ec0ea617b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/e0d3827a9520d9ca10ab2ff63ed96ab06dccaf98b37b6a2384ca73c8eff6a58a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/4630321b4602f2cbebf1915839e634e91c43ae9d41ce9fb9f0138f24bdefb841/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/2a8bd1490c1fb9b9ed555697ae5e22fea242397615f22fe31607a8f1a0e5d9a3/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5c38371d21c1cb270c7fbe4d51648ae3d1abc480a60964e512b8a02079179dca/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/0f3a9bd27dee59ab4e78c360ef36f7021ebd19dc90c0248ec42d7aaa6cd8d8d8/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/837a5717fedb2a29949ed87fc3010b7c91e775db68b07f7c41d4cbc3700a6911/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/2409f2c942fdb0728ea3907740ab6cb5132a106b88f19fb5b6e04a7e8afd0127/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/4eb69a1a8c15cf801106d2ec8531c035ae54c893d9492f6ca16a4b878c33734e/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/e6308dad354b4f5722634a87d21b179cda656f3ec0d28d46a95d6126128b9712/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/0fb7f913cb627be8befdeedd9a4c09bc0784ab0fedb8037d5c91643e375c9011/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/16f03d832341ce7b192b48c523676151a78d4506261909633304568cba69c6d8/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/b042bef7c13ed002209eb9d0a86cb6d035a77dbe317edfeb2069f46ccfbdd9bb/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f17e3b8486c03cdf01d760f5830599cf15de881264fe362a099ef96dc91b03a1/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/38c368a0b9cb31cf8c2be06962e55647ba14813776e30f82082c09b139bb2056/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/30ec79708498d85ecddd7df4bad554b44b6e2247d66aba1bf127e67ba416bb39/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/4772f8844aa53e4e50df0f061b04226317220491e1bb07dc3d3e6a05354ea6b4/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/a50b3ee22e8268290fa0e13d9565798a8826ac5aa0ce251df12c4725b7a26cff/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/216a8cd4de7cb4defda18a4765418ce427bb9493a239d06df9a7d3fdf988d32e/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/8e945775639b3f4fec753539521e6220151454122abb64f50821b62dc980f84d/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8e6da5f8f286748cae7ee143bc698f58c3576247f4333f5f8a258e437f641eb1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8cf1625474bd9250f6f318b5af2a6248200457d4edcd83d8ad34cdf1463d3091/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/ea083e5dbeceb87a3931770a2b84239e0cfc715f26a014922ef1df056be98e81/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/b6b92d9bac93a4e417bc8ff6ed03746a2ce5c8c632371477d4c6f07094c66414/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/926e542d6e752d0eac1c41d2d6febb284fc2f974c321253ae54f06df1a6330b3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c5defd1996b567b5c9c7e65169ff7cbaaa05ac139f8cbaa22b70b6129d27bfca/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/652ace7f86bddaf922a3434396471ec61078308ebd79b17be742878713ece8af/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3c2587fa4ad2daa60ffad036ee3e4caffd0a55c6e0295c9e015aa4d7ae8d55fd/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1add5c8044ebcc72977a18d1198374f620d7247e4ecb3ecbf65837da39e46151/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009140 4862 manager.go:217] Machine: {Timestamp:2026-10-30 17:58:21.008692347 +0000 UTC m=+0.057737251 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099740 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27f2bdfb8cb239bbb22f16f0d990a0 SystemUUID:ec27f2bd-fb8c-b239-bbb2-2f16f0d990a0 BootID:c3fce4a0-5e77-409a-9298-b43e8785ded3 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b5e199e-2896-450f-9e73-838fb7e12e48/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0311e96784e88a5761844fee94b5e902487d37e562f249063e7a567a5c866402/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4606c18ee91f4eddbb933f9a86249a4e84f54314c3523d8f76e1db8321eb2f5a/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/66e91a23-8415-48a9-924c-64f752b59272/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40/volumes/kubernetes.io~projected/kube-api-access-cg7gv DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07ea06b86bef3a20ce9115a13047804abe6ed2b3f536afcb024421693455c5aa/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/df983796-b36d-40f0-ae14-93ba96e518e3/volumes/kubernetes.io~projected/kube-api-access-6fc78 DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/66e91a23-8415-48a9-924c-64f752b59272/volumes/kubernetes.io~projected/kube-api-access-59rfw DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79402ab-1f1f-458b-b7c8-3c8209e69efb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65a1ff50dc1d7c32bf2b223fd3cf3bb7daa2be665fdec8fd3cc474af1a0fc815/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba92e9d064ca58f324001642abb2d2f5ae8fdb9b3acb7feb20f0f7a85ff9c522/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/874e6413-424d-4443-80fc-f9e4d0a63b6c/volumes/kubernetes.io~projected/kube-api-access-9mklf DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1bc68151-0620-48d2-a803-e7cb588c5b76/volumes/kubernetes.io~projected/kube-api-access-4zm6w DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/476cffce-abe2-409d-9b4a-792af41d8f1c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/529241e2-eb4e-4412-ad1f-37dee4aa9b65/volumes/kubernetes.io~projected/kube-api-access-bnl8d DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68b04e89e205b231b8c90f34b32b9d4bd6b1886358851360bf8681a376f07893/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/66e91a23-8415-48a9-924c-64f752b59272/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79402ab-1f1f-458b-b7c8-3c8209e69efb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0bfbeb5b-8284-4324-a8d0-95d95d23dfc6/volumes/kubernetes.io~projected/kube-api-access-7hrbz DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7caaae72-bd7d-44b5-902e-099da2ef429b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72430c9337e74812ba6b2ad90d827aa84f53a792941dbda4e33fd28e92f264fa/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e212f4ef613c4cf68f2ead0714d8b305f4eaa39361270349e41c22b0cb99416/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a/volumes/kubernetes.io~projected/kube-api-access-5g8s9 DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d79402ab-1f1f-458b-b7c8-3c8209e69efb/volumes/kubernetes.io~projected/kube-api-access-tlxpz DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e697292bee026df8571cc20abfa9fe1a8186b18c54128a40553fcbbc3fac837d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc6bbf56c484f44ef737360c94843ce9eb85de60ec726eeb3d15cfa53a262413/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96dd8351e8c09ab7b93fdf0d7494b9a33c1bd4c0b95c281ef9a60055a8c7341f/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abe3cbfd004f98d7f1a99e6dfda84ed966a56a5b4b4252334ff3f9b8fd6af697/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/50098568-1891-40d1-886a-9065582c2467/volumes/kubernetes.io~projected/kube-api-access-hlzws DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5b5e199e-2896-450f-9e73-838fb7e12e48/volumes/kubernetes.io~projected/kube-api-access-k6zgv DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa3a9cce436bca035669a359cef965751407914479af843ddae0087d5a983ecd/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b518a841587144a85ff55bd0dfc07e01e250dec1d1cc732ba4299f848eaf0ea6/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95855e4c59040c66b37ef72a54cb9de5487c85997d619f22c84accec198ab5be/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4f34929e-0b21-4743-af11-1360d39f162b/volumes/kubernetes.io~projected/kube-api-access-p7fzn DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b810f479-5f8c-4acc-89de-a41d288c8e2c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5b5e199e-2896-450f-9e73-838fb7e12e48/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/21caa3cf-ca4f-424a-9f40-6f91a7b176c3/volumes/kubernetes.io~projected/kube-api-access-cz7ww DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~projected/kube-api-access-bx4ss DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7/volumes/kubernetes.io~projected/kube-api-access-xlqs4 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/56c4dbcb-db0a-4e7d-a633-3ff9304037d4/volumes/kubernetes.io~projected/kube-api-access-dgvqb DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f11682e3-9202-425f-af04-da7b7c2351be/volumes/kubernetes.io~projected/kube-api-access-tj8ms DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fa9ca6a85eee5063651cc30465c5ce6eb0a41fedee0aff53c70c611dde27808/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c35677-9ec2-4c68-938f-a9cfcc872f79/volumes/kubernetes.io~projected/kube-api-access-pvzmq DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2558988e914b7ff0ecf6b2ad7c05113d298e2106d7f045367df2267c79f09499/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c598714d-0188-42bd-80e6-d9873dc2ae20/volumes/kubernetes.io~projected/kube-api-access-6dwv6 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/468096db-c6fe-46bd-b256-05f2132cee4a/volumes/kubernetes.io~projected/kube-api-access-flk52 DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd/volumes/kubernetes.io~projected/kube-api-access-q65zz DeviceMajor:0 DeviceMinor:310 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f4e39ad52d0ec41bac2e39cda30df3b4208773218386c5ffd2f44f991528f5a/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0654be5cdc8e86bc08fae1da65afa1855395fa4993dee17b4990e536079479a6/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/476cffce-abe2-409d-9b4a-792af41d8f1c/volumes/kubernetes.io~projected/kube-api-access-4lhrg DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes/kubernetes.io~projected/kube-api-access-cl9jp DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a8820c7-364d-4f01-90e2-ea9979976e47/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8d8c65293344bf97397e4e9f6d4fd6e5b7cde9e3f4bd5fe84369abf0200ac5d/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f619d36-116d-4434-bf1e-1aecb4d7fefe/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e0b6fd527e592f71ba6e70227f28f619dcdcb162b4ff7b24db493bec7884b17/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a78c392cc5757ee37c6a65b839638dc72ac734944b3661dc542cb126d0044288/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b810f479-5f8c-4acc-89de-a41d288c8e2c/volumes/kubernetes.io~projected/kube-api-access-hrnqk DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7caaae72-bd7d-44b5-902e-099da2ef429b/volumes/kubernetes.io~projected/kube-api-access-fwrhb DeviceMajor:0 DeviceMinor:237 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab36263f-c154-4f9f-9a8f-10fc38405e52/volumes/kubernetes.io~projected/kube-api-access-n7qcx DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/21caa3cf-ca4f-424a-9f40-6f91a7b176c3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a311faf-5bcd-47ab-b102-62aa0abf168c/volumes/kubernetes.io~projected/kube-api-access-mp5lz DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2558988e914b7ff MacAddress:2a:34:bc:3f:09:25 Speed:10000 Mtu:8901} {Name:4606c18ee91f4ed MacAddress:8e:ad:32:20:d4:f8 Speed:10000 Mtu:8901} {Name:68b04e89e205b23 MacAddress:0e:c9:1e:df:0a:cf Speed:10000 Mtu:8901} {Name:72430c9337e7481 MacAddress:5e:d3:90:b3:14:49 Speed:10000 Mtu:8901} {Name:7fa9ca6a85eee50 MacAddress:5a:2f:36:76:4b:15 Speed:10000 Mtu:8901} {Name:95855e4c59040c6 MacAddress:ca:27:24:e6:a1:4c Speed:10000 Mtu:8901} {Name:96dd8351e8c09ab MacAddress:ae:35:8a:37:df:6b Speed:10000 Mtu:8901} {Name:abe3cbfd004f98d MacAddress:22:77:e2:38:0d:03 Speed:10000 Mtu:8901} {Name:b518a841587144a MacAddress:86:b6:ec:3d:dd:6b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:de:24:9a:ad Speed:0 Mtu:9001} {Name:br-int MacAddress:c2:17:c8:3c:bb:73 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:de:24:9a:ad Speed:0 Mtu:9001} {Name:f8d8c65293344bf MacAddress:26:a9:a0:ec:79:ee Speed:10000 Mtu:8901} {Name:fa3a9cce436bca0 MacAddress:a2:ab:33:6e:4f:6a Speed:10000 Mtu:8901} {Name:fc6bbf56c484f44 MacAddress:1a:c4:5b:7a:31:4e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ca:07:74:41:b2:41 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:12:69:a6:2b:fc:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009486 4862 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009527 4862 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009792 4862 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009816 4862 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-246.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009927 4862 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009935 4862 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009944 4862 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009964 4862 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.009981 4862 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.010126 4862 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.010230 4862 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.010284 4862 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.010298 4862 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.010309 4862 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.010317 4862 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:58:21.011005 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.010325 4862 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:58:21.011717 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.011168 4862 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:58:21.011717 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.011181 4862 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:58:21.011717 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.011656 4862 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:58:21.013898 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.013881 4862 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:58:21.013898 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.013895 4862 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:58:21.014050 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014018 4862 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:58:21.014252 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014236 4862 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:58:21.014437 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014293 4862 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:58:21.014493 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014435 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:58:21.014493 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014456 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:58:21.014493 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014467 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:58:21.014493 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014476 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:58:21.014493 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014485 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:58:21.014493 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014494 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:58:21.014817 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014504 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:58:21.014817 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014513 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:58:21.014817 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014525 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:58:21.014817 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014533 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:58:21.014817 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014554 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:58:21.014817 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014569 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:58:21.014817 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014639 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:58:21.014817 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.014650 4862 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:58:21.018160 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.018144 4862 server.go:1263] "Started kubelet" Oct 30 17:58:21.018660 ip-10-0-128-246 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:58:21.019066 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.018987 4862 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:58:21.019221 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.019107 4862 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:58:21.019459 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.019370 4862 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:58:21.019861 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.019798 4862 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:58:21.020768 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.020752 4862 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:58:21.024170 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.024150 4862 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.024464 4862 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.024483 4862 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.024502 4862 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.024526 4862 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.024571 4862 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:52:42 +0000 UTC" deadline="2028-06-28 22:37:43.838687581 +0000 UTC" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.024586 4862 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14572h39m22.814103962s" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:21.024623 4862 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.026729 4862 factory.go:55] Registering systemd factory Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.026745 4862 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027223 4862 factory.go:153] Registering CRI-O factory Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027240 4862 factory.go:223] Registration of the crio container factory successfully Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027329 4862 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027320 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" volumeName="kubernetes.io/projected/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-kube-api-access-cz7ww" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027347 4862 factory.go:103] Registering Raw factory Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027350 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="529241e2-eb4e-4412-ad1f-37dee4aa9b65" volumeName="kubernetes.io/empty-dir/529241e2-eb4e-4412-ad1f-37dee4aa9b65-tmp-dir" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027361 4862 manager.go:1199] Started watching for new ooms in manager Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027365 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" volumeName="kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027379 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b5e199e-2896-450f-9e73-838fb7e12e48" volumeName="kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-default-certificate" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027390 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66e91a23-8415-48a9-924c-64f752b59272" volumeName="kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-tmp" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027404 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f619d36-116d-4434-bf1e-1aecb4d7fefe" volumeName="kubernetes.io/secret/7f619d36-116d-4434-bf1e-1aecb4d7fefe-agent-certs" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027416 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476cffce-abe2-409d-9b4a-792af41d8f1c" volumeName="kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-snapshots" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027429 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874e6413-424d-4443-80fc-f9e4d0a63b6c" volumeName="kubernetes.io/projected/874e6413-424d-4443-80fc-f9e4d0a63b6c-kube-api-access-9mklf" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027443 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93028940-79f3-45fa-be9b-6ddd8dab5633" volumeName="kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027456 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93028940-79f3-45fa-be9b-6ddd8dab5633" volumeName="kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027468 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e4d7d6-f980-4b03-8200-d6c5d8d3dc40" volumeName="kubernetes.io/projected/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-kube-api-access-cg7gv" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027484 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9f0fe3-754d-4da0-becb-f5484b5ab9dd" volumeName="kubernetes.io/empty-dir/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-data-volume" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027497 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79402ab-1f1f-458b-b7c8-3c8209e69efb" volumeName="kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-env-overrides" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027510 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c35677-9ec2-4c68-938f-a9cfcc872f79" volumeName="kubernetes.io/projected/d8c35677-9ec2-4c68-938f-a9cfcc872f79-kube-api-access-pvzmq" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027532 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" volumeName="kubernetes.io/secret/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-serving-cert" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027548 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476cffce-abe2-409d-9b4a-792af41d8f1c" volumeName="kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-service-ca-bundle" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027563 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" volumeName="kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-binary-copy" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027581 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7caaae72-bd7d-44b5-902e-099da2ef429b" volumeName="kubernetes.io/projected/7caaae72-bd7d-44b5-902e-099da2ef429b-kube-api-access-fwrhb" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027596 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93028940-79f3-45fa-be9b-6ddd8dab5633" volumeName="kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027613 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99e4d7d6-f980-4b03-8200-d6c5d8d3dc40" volumeName="kubernetes.io/empty-dir/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-tmp-dir" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027627 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="468096db-c6fe-46bd-b256-05f2132cee4a" volumeName="kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027642 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="748c9d09-9db4-4e7d-9389-1b74c8e0e2c7" volumeName="kubernetes.io/configmap/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027656 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab36263f-c154-4f9f-9a8f-10fc38405e52" volumeName="kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-daemon-config" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027682 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056042ce-b312-4d54-9f83-15cd37aa97ce" volumeName="kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027701 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bfbeb5b-8284-4324-a8d0-95d95d23dfc6" volumeName="kubernetes.io/projected/0bfbeb5b-8284-4324-a8d0-95d95d23dfc6-kube-api-access-7hrbz" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027719 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476cffce-abe2-409d-9b4a-792af41d8f1c" volumeName="kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027738 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476cffce-abe2-409d-9b4a-792af41d8f1c" volumeName="kubernetes.io/projected/476cffce-abe2-409d-9b4a-792af41d8f1c-kube-api-access-4lhrg" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027754 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a" volumeName="kubernetes.io/projected/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-kube-api-access-5g8s9" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027781 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93028940-79f3-45fa-be9b-6ddd8dab5633" volumeName="kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027800 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b810f479-5f8c-4acc-89de-a41d288c8e2c" volumeName="kubernetes.io/configmap/b810f479-5f8c-4acc-89de-a41d288c8e2c-config" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027812 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c598714d-0188-42bd-80e6-d9873dc2ae20" volumeName="kubernetes.io/projected/c598714d-0188-42bd-80e6-d9873dc2ae20-kube-api-access-6dwv6" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027827 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056042ce-b312-4d54-9f83-15cd37aa97ce" volumeName="kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027841 4862 manager.go:319] Starting recovery of all containers Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027841 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476cffce-abe2-409d-9b4a-792af41d8f1c" volumeName="kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-tmp" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027863 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="529241e2-eb4e-4412-ad1f-37dee4aa9b65" volumeName="kubernetes.io/projected/529241e2-eb4e-4412-ad1f-37dee4aa9b65-kube-api-access-bnl8d" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027878 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" volumeName="kubernetes.io/projected/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-kube-api-access-dgvqb" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027890 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b5e199e-2896-450f-9e73-838fb7e12e48" volumeName="kubernetes.io/projected/5b5e199e-2896-450f-9e73-838fb7e12e48-kube-api-access-k6zgv" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027905 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66e91a23-8415-48a9-924c-64f752b59272" volumeName="kubernetes.io/projected/66e91a23-8415-48a9-924c-64f752b59272-kube-api-access-59rfw" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027916 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f619d36-116d-4434-bf1e-1aecb4d7fefe" volumeName="kubernetes.io/configmap/7f619d36-116d-4434-bf1e-1aecb4d7fefe-konnectivity-ca" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027927 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c35677-9ec2-4c68-938f-a9cfcc872f79" volumeName="kubernetes.io/configmap/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-config" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027941 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7caaae72-bd7d-44b5-902e-099da2ef429b" volumeName="kubernetes.io/secret/7caaae72-bd7d-44b5-902e-099da2ef429b-cert" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027952 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874e6413-424d-4443-80fc-f9e4d0a63b6c" volumeName="kubernetes.io/configmap/874e6413-424d-4443-80fc-f9e4d0a63b6c-telemetry-config" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027967 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93028940-79f3-45fa-be9b-6ddd8dab5633" volumeName="kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027978 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df983796-b36d-40f0-ae14-93ba96e518e3" volumeName="kubernetes.io/projected/df983796-b36d-40f0-ae14-93ba96e518e3-kube-api-access-6fc78" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.027989 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" volumeName="kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-trusted-ca" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028001 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f34929e-0b21-4743-af11-1360d39f162b" volumeName="kubernetes.io/configmap/4f34929e-0b21-4743-af11-1360d39f162b-serviceca" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028013 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79402ab-1f1f-458b-b7c8-3c8209e69efb" volumeName="kubernetes.io/projected/d79402ab-1f1f-458b-b7c8-3c8209e69efb-kube-api-access-tlxpz" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028026 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79402ab-1f1f-458b-b7c8-3c8209e69efb" volumeName="kubernetes.io/secret/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028040 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="529241e2-eb4e-4412-ad1f-37dee4aa9b65" volumeName="kubernetes.io/configmap/529241e2-eb4e-4412-ad1f-37dee4aa9b65-config-volume" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028051 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056042ce-b312-4d54-9f83-15cd37aa97ce" volumeName="kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028067 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a" volumeName="kubernetes.io/configmap/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-config" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028081 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b810f479-5f8c-4acc-89de-a41d288c8e2c" volumeName="kubernetes.io/projected/b810f479-5f8c-4acc-89de-a41d288c8e2c-kube-api-access-hrnqk" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028094 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056042ce-b312-4d54-9f83-15cd37aa97ce" volumeName="kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028107 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056042ce-b312-4d54-9f83-15cd37aa97ce" volumeName="kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028119 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f34929e-0b21-4743-af11-1360d39f162b" volumeName="kubernetes.io/projected/4f34929e-0b21-4743-af11-1360d39f162b-kube-api-access-p7fzn" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028131 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93028940-79f3-45fa-be9b-6ddd8dab5633" volumeName="kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028143 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93028940-79f3-45fa-be9b-6ddd8dab5633" volumeName="kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028168 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b810f479-5f8c-4acc-89de-a41d288c8e2c" volumeName="kubernetes.io/secret/b810f479-5f8c-4acc-89de-a41d288c8e2c-serving-cert" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028182 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79402ab-1f1f-458b-b7c8-3c8209e69efb" volumeName="kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-config" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028194 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056042ce-b312-4d54-9f83-15cd37aa97ce" volumeName="kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028207 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8820c7-364d-4f01-90e2-ea9979976e47" volumeName="kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028221 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab36263f-c154-4f9f-9a8f-10fc38405e52" volumeName="kubernetes.io/projected/ab36263f-c154-4f9f-9a8f-10fc38405e52-kube-api-access-n7qcx" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028232 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" volumeName="kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028242 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce" volumeName="kubernetes.io/configmap/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-nginx-conf" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028256 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9f0fe3-754d-4da0-becb-f5484b5ab9dd" volumeName="kubernetes.io/projected/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-api-access-q65zz" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028285 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056042ce-b312-4d54-9f83-15cd37aa97ce" volumeName="kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028298 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" volumeName="kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-config" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028310 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a" volumeName="kubernetes.io/secret/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-serving-cert" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028321 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b5e199e-2896-450f-9e73-838fb7e12e48" volumeName="kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-stats-auth" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028332 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66e91a23-8415-48a9-924c-64f752b59272" volumeName="kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-etc-tuned" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028343 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bc68151-0620-48d2-a803-e7cb588c5b76" volumeName="kubernetes.io/projected/1bc68151-0620-48d2-a803-e7cb588c5b76-kube-api-access-4zm6w" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028353 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="476cffce-abe2-409d-9b4a-792af41d8f1c" volumeName="kubernetes.io/secret/476cffce-abe2-409d-9b4a-792af41d8f1c-serving-cert" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028365 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a311faf-5bcd-47ab-b102-62aa0abf168c" volumeName="kubernetes.io/projected/5a311faf-5bcd-47ab-b102-62aa0abf168c-kube-api-access-mp5lz" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028376 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="748c9d09-9db4-4e7d-9389-1b74c8e0e2c7" volumeName="kubernetes.io/projected/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-kube-api-access-xlqs4" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028388 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab36263f-c154-4f9f-9a8f-10fc38405e52" volumeName="kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-cni-binary-copy" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028399 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f11682e3-9202-425f-af04-da7b7c2351be" volumeName="kubernetes.io/projected/f11682e3-9202-425f-af04-da7b7c2351be-kube-api-access-tj8ms" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028410 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50098568-1891-40d1-886a-9065582c2467" volumeName="kubernetes.io/projected/50098568-1891-40d1-886a-9065582c2467-kube-api-access-hlzws" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028421 4862 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d79402ab-1f1f-458b-b7c8-3c8209e69efb" volumeName="kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028435 4862 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:58:21.028713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.028442 4862 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:58:21.045149 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.045129 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/1.log" Oct 30 17:58:21.045832 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.045675 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/1.log" Oct 30 17:58:21.052610 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.052567 4862 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:58:21.053931 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.053910 4862 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:58:21.054007 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.053934 4862 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:58:21.054007 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.053959 4862 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:58:21.054007 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:21.053999 4862 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:58:21.059175 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.059157 4862 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:21.069550 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.069486 4862 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:58:21.080428 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.080364 4862 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:21.080807 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.080787 4862 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:21.080903 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.080862 4862 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:58:21.081581 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.081535 4862 generic.go:356] "Generic (PLEG): container finished" podID="67318722ef4c3e2a1b8bc7b38d57c5ad" containerID="e741c9bb968530dc0edbdae04b72a794b5b162deb5c0bec13646df0f86b8399b" exitCode=0 Oct 30 17:58:21.093488 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.093464 4862 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="5a0078c8f530cbe1f0d5871b8e830187e4b95c7e49f279b6aa3d2a8c7c729265" exitCode=0 Oct 30 17:58:21.093488 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.093485 4862 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="17b9964111f6c0842c35f80c4751fc81c574460ad498e19b8bfddcce28975def" exitCode=0 Oct 30 17:58:21.093607 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.093494 4862 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="ffde1b0388f82041b371b5e5396324962661d1898eb7a3f36eb79b817f9d9529" exitCode=0 Oct 30 17:58:21.093607 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.093500 4862 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="6024365945f9cb716064178b533f05a0c6d709e025eb02d01f83312a42a45ba5" exitCode=0 Oct 30 17:58:21.093607 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.093506 4862 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="9e8495a0f874548a99e2cd72fb5c24a9c38a991dfc14c6fcffb2acea9bff0ac0" exitCode=0 Oct 30 17:58:21.093607 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.093511 4862 generic.go:356] "Generic (PLEG): container finished" podID="56c4dbcb-db0a-4e7d-a633-3ff9304037d4" containerID="4adeba7a8f3b1f39480e52fecacd67106e3df7e524100bae9ba3f40b7b75d499" exitCode=0 Oct 30 17:58:21.100969 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.100953 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/1.log" Oct 30 17:58:21.101060 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.100981 4862 generic.go:356] "Generic (PLEG): container finished" podID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" containerID="d3ca2364d68fe0246dca3420e6db36a2937baca9ad9ef1d17b53de359919a379" exitCode=255 Oct 30 17:58:21.114563 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.114516 4862 manager.go:324] Recovery completed Oct 30 17:58:21.133659 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133642 4862 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:58:21.133659 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133654 4862 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:58:21.133752 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133669 4862 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:58:21.133794 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133775 4862 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:58:21.133823 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133785 4862 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:58:21.133823 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133801 4862 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:58:21.133823 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133806 4862 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:58:21.133823 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133811 4862 policy_none.go:50] "Start" Oct 30 17:58:21.133823 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133819 4862 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:58:21.133950 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133829 4862 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:58:21.133984 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133960 4862 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:58:21.133984 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133967 4862 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:58:21.133984 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.133972 4862 policy_none.go:44] "Start" Oct 30 17:58:21.138106 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.138083 4862 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:58:21.140581 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.140569 4862 manager.go:341] "Starting Device Plugin manager" Oct 30 17:58:21.141093 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:21.140592 4862 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:58:21.141093 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.141095 4862 server.go:87] "Starting device plugin registration server" Oct 30 17:58:21.141349 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.141337 4862 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:58:21.141402 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.141348 4862 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:58:21.141463 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.141449 4862 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:58:21.141562 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.141534 4862 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:58:21.141675 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.141566 4862 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:58:21.141675 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.141573 4862 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:58:21.141675 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.141614 4862 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:58:21.141556615Z","UUID":"3d0dd987-7056-4ef9-b40f-e8d8a57510af","Handler":null,"Name":"","Endpoint":""} Oct 30 17:58:21.142755 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:21.142736 4862 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:58:21.142986 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.142972 4862 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:58:21.143042 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.142993 4862 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:58:21.154076 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.154039 4862 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal"] Oct 30 17:58:21.154703 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.154688 4862 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.154759 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.154720 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" event={"ID":"46fe009f9a7f834e879e5480aab152d6","Type":"ContainerStarted","Data":"33035195870ba00fd49df006b0f69ed4068c47a0e80b4a2cdbf12e0807f8caa4"} Oct 30 17:58:21.154793 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.154758 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" event={"ID":"46fe009f9a7f834e879e5480aab152d6","Type":"ContainerStarted","Data":"7f4e39ad52d0ec41bac2e39cda30df3b4208773218386c5ffd2f44f991528f5a"} Oct 30 17:58:21.154793 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.154776 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" event={"ID":"67318722ef4c3e2a1b8bc7b38d57c5ad","Type":"ContainerStarted","Data":"45e0842ad0c9063ea6c307498973e1bba051082e0059aa79fd3594103661bbb8"} Oct 30 17:58:21.154847 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.154794 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" event={"ID":"67318722ef4c3e2a1b8bc7b38d57c5ad","Type":"ContainerDied","Data":"e741c9bb968530dc0edbdae04b72a794b5b162deb5c0bec13646df0f86b8399b"} Oct 30 17:58:21.154847 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.154815 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" event={"ID":"67318722ef4c3e2a1b8bc7b38d57c5ad","Type":"ContainerStarted","Data":"0311e96784e88a5761844fee94b5e902487d37e562f249063e7a567a5c866402"} Oct 30 17:58:21.154979 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.154965 4862 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.180079 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.180060 4862 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:21.180152 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:21.180107 4862 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.181685 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.181671 4862 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:21.181731 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:21.181706 4862 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.242456 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.242440 4862 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:58:21.243471 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.243458 4862 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:58:21.243524 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.243484 4862 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:58:21.243524 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.243494 4862 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:58:21.243580 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.243528 4862 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.270542 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.270523 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.270617 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.270554 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.270617 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.270573 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46fe009f9a7f834e879e5480aab152d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" (UID: \"46fe009f9a7f834e879e5480aab152d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.279179 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.279162 4862 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.279236 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.279209 4862 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.371522 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.371457 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.371522 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.371477 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.371660 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.371528 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.371660 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.371546 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46fe009f9a7f834e879e5480aab152d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" (UID: \"46fe009f9a7f834e879e5480aab152d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.371660 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.371580 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/46fe009f9a7f834e879e5480aab152d6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" (UID: \"46fe009f9a7f834e879e5480aab152d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:58:21.371660 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:21.371625 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67318722ef4c3e2a1b8bc7b38d57c5ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"67318722ef4c3e2a1b8bc7b38d57c5ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:22.010790 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.010761 4862 apiserver.go:52] "Watching apiserver" Oct 30 17:58:22.067681 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.067652 4862 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:58:22.069920 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.069878 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-hk9ss","openshift-multus/multus-ch8b8","openshift-network-operator/iptables-alerter-j8t4c","openshift-dns/dns-default-r94b8","openshift-image-registry/image-registry-74f5c5549c-bgqzt","openshift-ingress/router-default-ff9667845-fjrpd","kube-system/konnectivity-agent-vz7gh","openshift-cluster-node-tuning-operator/tuned-zb8gk","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j","openshift-insights/insights-operator-7c4986d449-zg24h","openshift-network-diagnostics/network-check-target-q74cr","kube-system/kas-connection-checker-685895c7c9-qnth8","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m","openshift-image-registry/image-registry-55dcc4fcc6-4mw7c","openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz","kube-system/global-pull-secret-syncer-dt49r","openshift-image-registry/node-ca-6wlg8","kube-system/kas-connection-checker-685895c7c9-6dk47","openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv","kube-system/kas-connection-checker-685895c7c9-6cmjc","openshift-ovn-kubernetes/ovnkube-node-dx2gw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm","openshift-console-operator/console-operator-7cd56b8dbc-gxwj6","kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal","openshift-dns/node-resolver-rv2ks","openshift-ingress-canary/ingress-canary-s2zkr","openshift-network-console/networking-console-plugin-86c478bb88-mk86v","openshift-multus/multus-additional-cni-plugins-p4pv4","openshift-insights/insights-runtime-extractor-f2zht"] Oct 30 17:58:22.072099 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.072081 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:22.072391 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.072366 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.072801 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.072775 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.074802 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.074785 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:22.075401 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.075387 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:22.080678 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.080661 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:22.081780 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.081765 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.082921 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.082906 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.083377 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.083360 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.087457 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.087440 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.087583 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.087440 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.088359 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.088348 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.094465 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.094446 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.094559 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.094483 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 17:58:22.094614 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.094556 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 17:58:22.094989 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.094975 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 17:58:22.095252 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.095233 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 17:58:22.095354 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.095297 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:58:22.095461 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.095448 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.095511 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.095475 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cv2gk\"" Oct 30 17:58:22.095835 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.095822 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zdsf4\"" Oct 30 17:58:22.096433 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.096419 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 17:58:22.097765 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.097753 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 17:58:22.104127 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.104113 4862 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:58:22.104227 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.104215 4862 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:22.106295 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106255 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 17:58:22.106438 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106422 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 17:58:22.106529 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106515 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 17:58:22.106593 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106573 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzq26\"" Oct 30 17:58:22.106678 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106603 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9m88r\"" Oct 30 17:58:22.106678 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106619 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:58:22.106678 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106629 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:58:22.106678 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106660 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:58:22.106852 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106680 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 17:58:22.106852 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106697 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-c6ggl\"" Oct 30 17:58:22.106852 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106704 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:58:22.106852 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106710 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 17:58:22.106852 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106726 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 17:58:22.106852 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106779 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:58:22.106852 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106795 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6fpb9\"" Oct 30 17:58:22.106852 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106806 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 17:58:22.106852 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.106820 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xl6hf\"" Oct 30 17:58:22.107112 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107032 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 17:58:22.107112 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107047 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8mdkh\"" Oct 30 17:58:22.107112 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107054 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dltg8\"" Oct 30 17:58:22.107330 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107311 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p48xz\"" Oct 30 17:58:22.107330 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107311 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt2qr\"" Oct 30 17:58:22.107509 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107398 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jcq9r\"" Oct 30 17:58:22.107509 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107408 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nfj27\"" Oct 30 17:58:22.107509 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107414 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zdwtx\"" Oct 30 17:58:22.107648 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107519 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh48h\"" Oct 30 17:58:22.107648 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107537 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9dlsp\"" Oct 30 17:58:22.107648 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107540 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 17:58:22.107648 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107582 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:58:22.107648 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107601 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hhtgn\"" Oct 30 17:58:22.107648 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107609 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2xlsx\"" Oct 30 17:58:22.107648 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107604 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6627\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107659 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107733 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107744 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-swnsr\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107756 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdnw2\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107779 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wj5nd\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107787 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nwsqb\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107812 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107814 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mngtd\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107815 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107878 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pn5pr\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107881 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.107946 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107938 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qgvl5\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.107999 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108045 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108070 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108097 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108097 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108113 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108163 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108171 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108170 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108188 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108201 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108225 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108226 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:58:22.108427 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108240 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:58:22.108951 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108472 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:58:22.108951 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108492 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.108951 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108496 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.108951 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.108506 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 17:58:22.109619 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.109605 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.109678 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.109645 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 17:58:22.109734 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.109688 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.109734 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.109689 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.109734 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.109728 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.109862 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.109797 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 17:58:22.109862 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.109824 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:58:22.109925 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.109861 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.116040 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.116022 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:58:22.116124 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.116055 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.116124 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.116057 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.116124 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.116107 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.116124 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.116117 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.116124 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.116123 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.116298 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.116124 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.116298 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.116107 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.117116 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117100 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.117116 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117112 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:58:22.117247 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117103 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.117247 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117134 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.117247 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117142 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:58:22.117247 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117193 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.117247 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117211 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:58:22.117247 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117227 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.117502 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117280 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:58:22.117502 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117283 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:58:22.117502 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117291 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:58:22.117502 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117447 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:58:22.117865 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.117852 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 17:58:22.118542 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.118529 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 17:58:22.126010 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.125969 4862 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:22.126062 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.126017 4862 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" Oct 30 17:58:22.126984 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.126971 4862 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:58:22.130598 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.130584 4862 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:58:22.130646 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.130623 4862 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" Oct 30 17:58:22.175043 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175006 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-systemd\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.175043 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175039 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-kubelet-config\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:22.175226 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175059 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cnibin\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.175226 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175080 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/874e6413-424d-4443-80fc-f9e4d0a63b6c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:22.175226 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175146 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-hostroot\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.175226 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175172 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.175401 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175230 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhrg\" (UniqueName: \"kubernetes.io/projected/476cffce-abe2-409d-9b4a-792af41d8f1c-kube-api-access-4lhrg\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.175401 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175306 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.175480 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175404 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mklf\" (UniqueName: \"kubernetes.io/projected/874e6413-424d-4443-80fc-f9e4d0a63b6c-kube-api-access-9mklf\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:22.175480 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175422 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7gv\" (UniqueName: \"kubernetes.io/projected/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-kube-api-access-cg7gv\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:58:22.175480 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175423 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.175618 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175497 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7ww\" (UniqueName: \"kubernetes.io/projected/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-kube-api-access-cz7ww\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:22.175618 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175489 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/874e6413-424d-4443-80fc-f9e4d0a63b6c-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:22.175618 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175562 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.175618 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175591 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-iptables-alerter-script\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:58:22.175618 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175613 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:22.175825 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175648 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-systemd-units\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.175825 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175671 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.175825 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175690 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-bin\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.175825 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175719 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:22.175825 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175761 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zm6w\" (UniqueName: \"kubernetes.io/projected/1bc68151-0620-48d2-a803-e7cb588c5b76-kube-api-access-4zm6w\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:22.175825 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175807 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-stats-auth\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.175825 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175828 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7fzn\" (UniqueName: \"kubernetes.io/projected/4f34929e-0b21-4743-af11-1360d39f162b-kube-api-access-p7fzn\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175846 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwv6\" (UniqueName: \"kubernetes.io/projected/c598714d-0188-42bd-80e6-d9873dc2ae20-kube-api-access-6dwv6\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175858 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175862 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7caaae72-bd7d-44b5-902e-099da2ef429b-cert\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175889 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-system-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175913 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-config\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175938 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175949 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-iptables-alerter-script\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175958 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175968 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.175990 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-host-slash\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176029 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-conf-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176141 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-config\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176148 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-stats-auth\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.176162 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176152 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176186 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-default-certificate\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176208 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7caaae72-bd7d-44b5-902e-099da2ef429b-cert\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176232 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-lib-modules\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176314 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176351 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-cnibin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176363 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-trusted-ca\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176396 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-default-certificate\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176401 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-netns\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176437 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-multus-certs\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176461 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176479 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529241e2-eb4e-4412-ad1f-37dee4aa9b65-tmp-dir\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176495 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-etc-kubernetes\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176510 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176525 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-var-lib-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176540 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-script-lib\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176558 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxpz\" (UniqueName: \"kubernetes.io/projected/d79402ab-1f1f-458b-b7c8-3c8209e69efb-kube-api-access-tlxpz\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176571 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176624 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529241e2-eb4e-4412-ad1f-37dee4aa9b65-tmp-dir\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.176658 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176663 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-hosts-file\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176726 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176728 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176772 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/66e91a23-8415-48a9-924c-64f752b59272-kube-api-access-59rfw\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176792 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176816 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvqb\" (UniqueName: \"kubernetes.io/projected/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-kube-api-access-dgvqb\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176933 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-sys\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176938 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-script-lib\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176972 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-registration-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.176992 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnqk\" (UniqueName: \"kubernetes.io/projected/b810f479-5f8c-4acc-89de-a41d288c8e2c-kube-api-access-hrnqk\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177008 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-netns\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177053 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovn-node-metrics-cert\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177091 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177148 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnl8d\" (UniqueName: \"kubernetes.io/projected/529241e2-eb4e-4412-ad1f-37dee4aa9b65-kube-api-access-bnl8d\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177176 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-run\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177204 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f619d36-116d-4434-bf1e-1aecb4d7fefe-konnectivity-ca\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177229 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8ms\" (UniqueName: \"kubernetes.io/projected/f11682e3-9202-425f-af04-da7b7c2351be-kube-api-access-tj8ms\") pod \"kas-connection-checker-685895c7c9-6cmjc\" (UID: \"f11682e3-9202-425f-af04-da7b7c2351be\") " pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177245 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovn-node-metrics-cert\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177299 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-kubelet\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.177336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177331 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177364 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177375 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-netd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177397 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177433 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9jp\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177519 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7f619d36-116d-4434-bf1e-1aecb4d7fefe-konnectivity-ca\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177561 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177576 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-env-overrides\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177612 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-data-volume\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177639 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177663 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/476cffce-abe2-409d-9b4a-792af41d8f1c-serving-cert\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177674 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177694 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177719 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-env-overrides\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177721 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177739 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-data-volume\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177764 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-conf\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177847 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/476cffce-abe2-409d-9b4a-792af41d8f1c-serving-cert\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177870 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.177911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177902 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177926 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b810f479-5f8c-4acc-89de-a41d288c8e2c-serving-cert\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177966 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.177988 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178028 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178034 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b810f479-5f8c-4acc-89de-a41d288c8e2c-serving-cert\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178061 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-socket-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178104 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-etc-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178141 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178161 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178202 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/529241e2-eb4e-4412-ad1f-37dee4aa9b65-config-volume\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178220 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178229 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4ss\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178292 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178292 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-slash\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178330 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-socket-dir-parent\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178386 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-device-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.178430 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178421 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-kubelet\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178452 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-etc-tuned\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178474 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-bin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178490 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178490 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/529241e2-eb4e-4412-ad1f-37dee4aa9b65-config-volume\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178508 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hrbz\" (UniqueName: \"kubernetes.io/projected/0bfbeb5b-8284-4324-a8d0-95d95d23dfc6-kube-api-access-7hrbz\") pod \"volume-data-source-validator-58c9f5f46d-lkc5k\" (UID: \"0bfbeb5b-8284-4324-a8d0-95d95d23dfc6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178539 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-etc-tuned\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178570 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178598 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp5lz\" (UniqueName: \"kubernetes.io/projected/5a311faf-5bcd-47ab-b102-62aa0abf168c-kube-api-access-mp5lz\") pod \"kas-connection-checker-685895c7c9-qnth8\" (UID: \"5a311faf-5bcd-47ab-b102-62aa0abf168c\") " pod="kube-system/kas-connection-checker-685895c7c9-qnth8" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178704 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlqs4\" (UniqueName: \"kubernetes.io/projected/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-kube-api-access-xlqs4\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178728 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qcx\" (UniqueName: \"kubernetes.io/projected/ab36263f-c154-4f9f-9a8f-10fc38405e52-kube-api-access-n7qcx\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178745 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-snapshots\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178786 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178793 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178800 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-snapshots\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.178869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178829 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178951 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q65zz\" (UniqueName: \"kubernetes.io/projected/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-api-access-q65zz\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178979 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.178986 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179033 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179053 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fc78\" (UniqueName: \"kubernetes.io/projected/df983796-b36d-40f0-ae14-93ba96e518e3-kube-api-access-6fc78\") pod \"network-check-source-6f78d59f99-rrfpm\" (UID: \"df983796-b36d-40f0-ae14-93ba96e518e3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179068 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-config\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179082 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-tmp\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179101 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-modprobe-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179123 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-host\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179148 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179187 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/476cffce-abe2-409d-9b4a-792af41d8f1c-tmp\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179200 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/476cffce-abe2-409d-9b4a-792af41d8f1c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179202 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzmq\" (UniqueName: \"kubernetes.io/projected/d8c35677-9ec2-4c68-938f-a9cfcc872f79-kube-api-access-pvzmq\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179250 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179282 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d79402ab-1f1f-458b-b7c8-3c8209e69efb-ovnkube-config\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179377 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysconfig\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179457 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.179470 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179479 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179496 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179546 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-crio-socket\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179574 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179577 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179599 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179632 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f34929e-0b21-4743-af11-1360d39f162b-host\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179659 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179713 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-serving-cert\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179766 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-var-lib-kubelet\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179795 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179846 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-sys-fs\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179876 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-serving-cert\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179887 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-dbus\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179924 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-cni-binary-copy\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179951 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.179961 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180052 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8s9\" (UniqueName: \"kubernetes.io/projected/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-kube-api-access-5g8s9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180078 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-tmp\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.180102 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180105 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-ovn\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180133 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-daemon-config\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180158 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-cni-binary-copy\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180160 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180197 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-system-cni-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180201 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66e91a23-8415-48a9-924c-64f752b59272-tmp\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180248 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180315 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/5b5e199e-2896-450f-9e73-838fb7e12e48-kube-api-access-k6zgv\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180343 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-tmp-dir\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180347 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-daemon-config\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180434 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b810f479-5f8c-4acc-89de-a41d288c8e2c-config\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180443 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a8820c7-364d-4f01-90e2-ea9979976e47-original-pull-secret\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180461 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-log-socket\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180479 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlzws\" (UniqueName: \"kubernetes.io/projected/50098568-1891-40d1-886a-9065582c2467-kube-api-access-hlzws\") pod \"kas-connection-checker-685895c7c9-6dk47\" (UID: \"50098568-1891-40d1-886a-9065582c2467\") " pod="kube-system/kas-connection-checker-685895c7c9-6dk47" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180479 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-tmp-dir\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180508 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180596 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-systemd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180616 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b810f479-5f8c-4acc-89de-a41d288c8e2c-config\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180689 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrhb\" (UniqueName: \"kubernetes.io/projected/7caaae72-bd7d-44b5-902e-099da2ef429b-kube-api-access-fwrhb\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180714 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-multus\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180805 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f619d36-116d-4434-bf1e-1aecb4d7fefe-agent-certs\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:58:22.181538 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180864 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-node-log\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.181538 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180892 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.181538 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180919 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-k8s-cni-cncf-io\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.181538 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180945 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f34929e-0b21-4743-af11-1360d39f162b-serviceca\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:58:22.181538 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.180973 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-kubernetes\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.181538 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.181048 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7f619d36-116d-4434-bf1e-1aecb4d7fefe-agent-certs\") pod \"konnectivity-agent-vz7gh\" (UID: \"7f619d36-116d-4434-bf1e-1aecb4d7fefe\") " pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:58:22.181538 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.181109 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f34929e-0b21-4743-af11-1360d39f162b-serviceca\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:58:22.193367 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.193341 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhrg\" (UniqueName: \"kubernetes.io/projected/476cffce-abe2-409d-9b4a-792af41d8f1c-kube-api-access-4lhrg\") pod \"insights-operator-7c4986d449-zg24h\" (UID: \"476cffce-abe2-409d-9b4a-792af41d8f1c\") " pod="openshift-insights/insights-operator-7c4986d449-zg24h" Oct 30 17:58:22.196354 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196195 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnl8d\" (UniqueName: \"kubernetes.io/projected/529241e2-eb4e-4412-ad1f-37dee4aa9b65-kube-api-access-bnl8d\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.196354 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196244 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxpz\" (UniqueName: \"kubernetes.io/projected/d79402ab-1f1f-458b-b7c8-3c8209e69efb-kube-api-access-tlxpz\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.196354 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196288 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp5lz\" (UniqueName: \"kubernetes.io/projected/5a311faf-5bcd-47ab-b102-62aa0abf168c-kube-api-access-mp5lz\") pod \"kas-connection-checker-685895c7c9-qnth8\" (UID: \"5a311faf-5bcd-47ab-b102-62aa0abf168c\") " pod="kube-system/kas-connection-checker-685895c7c9-qnth8" Oct 30 17:58:22.196354 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196298 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwv6\" (UniqueName: \"kubernetes.io/projected/c598714d-0188-42bd-80e6-d9873dc2ae20-kube-api-access-6dwv6\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.196653 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196359 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7gv\" (UniqueName: \"kubernetes.io/projected/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-kube-api-access-cg7gv\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:58:22.196653 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196373 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7fzn\" (UniqueName: \"kubernetes.io/projected/4f34929e-0b21-4743-af11-1360d39f162b-kube-api-access-p7fzn\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:58:22.196653 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196575 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9jp\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.196810 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196654 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7ww\" (UniqueName: \"kubernetes.io/projected/21caa3cf-ca4f-424a-9f40-6f91a7b176c3-kube-api-access-cz7ww\") pod \"console-operator-7cd56b8dbc-gxwj6\" (UID: \"21caa3cf-ca4f-424a-9f40-6f91a7b176c3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:22.196902 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196880 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnqk\" (UniqueName: \"kubernetes.io/projected/b810f479-5f8c-4acc-89de-a41d288c8e2c-kube-api-access-hrnqk\") pod \"service-ca-operator-544d749c8c-wcpjz\" (UID: \"b810f479-5f8c-4acc-89de-a41d288c8e2c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" Oct 30 17:58:22.196970 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.196952 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flk52\" (UniqueName: \"kubernetes.io/projected/468096db-c6fe-46bd-b256-05f2132cee4a-kube-api-access-flk52\") pod \"network-check-target-q74cr\" (UID: \"468096db-c6fe-46bd-b256-05f2132cee4a\") " pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:22.197050 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.197025 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4ss\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.197125 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.197110 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rfw\" (UniqueName: \"kubernetes.io/projected/66e91a23-8415-48a9-924c-64f752b59272-kube-api-access-59rfw\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.197185 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.197130 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvqb\" (UniqueName: \"kubernetes.io/projected/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-kube-api-access-dgvqb\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.197299 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.197283 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zm6w\" (UniqueName: \"kubernetes.io/projected/1bc68151-0620-48d2-a803-e7cb588c5b76-kube-api-access-4zm6w\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:22.197355 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.197326 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hrbz\" (UniqueName: \"kubernetes.io/projected/0bfbeb5b-8284-4324-a8d0-95d95d23dfc6-kube-api-access-7hrbz\") pod \"volume-data-source-validator-58c9f5f46d-lkc5k\" (UID: \"0bfbeb5b-8284-4324-a8d0-95d95d23dfc6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lkc5k" Oct 30 17:58:22.197924 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.197909 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mklf\" (UniqueName: \"kubernetes.io/projected/874e6413-424d-4443-80fc-f9e4d0a63b6c-kube-api-access-9mklf\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:22.200327 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.200310 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qcx\" (UniqueName: \"kubernetes.io/projected/ab36263f-c154-4f9f-9a8f-10fc38405e52-kube-api-access-n7qcx\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.200706 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.200686 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlqs4\" (UniqueName: \"kubernetes.io/projected/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-kube-api-access-xlqs4\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:58:22.201526 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.201505 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.201690 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.201676 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65zz\" (UniqueName: \"kubernetes.io/projected/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-api-access-q65zz\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.201798 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.201777 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8ms\" (UniqueName: \"kubernetes.io/projected/f11682e3-9202-425f-af04-da7b7c2351be-kube-api-access-tj8ms\") pod \"kas-connection-checker-685895c7c9-6cmjc\" (UID: \"f11682e3-9202-425f-af04-da7b7c2351be\") " pod="kube-system/kas-connection-checker-685895c7c9-6cmjc" Oct 30 17:58:22.201917 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.201903 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fc78\" (UniqueName: \"kubernetes.io/projected/df983796-b36d-40f0-ae14-93ba96e518e3-kube-api-access-6fc78\") pod \"network-check-source-6f78d59f99-rrfpm\" (UID: \"df983796-b36d-40f0-ae14-93ba96e518e3\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rrfpm" Oct 30 17:58:22.204444 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.204428 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.204679 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.204658 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlzws\" (UniqueName: \"kubernetes.io/projected/50098568-1891-40d1-886a-9065582c2467-kube-api-access-hlzws\") pod \"kas-connection-checker-685895c7c9-6dk47\" (UID: \"50098568-1891-40d1-886a-9065582c2467\") " pod="kube-system/kas-connection-checker-685895c7c9-6dk47" Oct 30 17:58:22.204711 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.204699 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzmq\" (UniqueName: \"kubernetes.io/projected/d8c35677-9ec2-4c68-938f-a9cfcc872f79-kube-api-access-pvzmq\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:22.205932 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.205918 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zgv\" (UniqueName: \"kubernetes.io/projected/5b5e199e-2896-450f-9e73-838fb7e12e48-kube-api-access-k6zgv\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.206174 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.206156 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrhb\" (UniqueName: \"kubernetes.io/projected/7caaae72-bd7d-44b5-902e-099da2ef429b-kube-api-access-fwrhb\") pod \"ingress-canary-s2zkr\" (UID: \"7caaae72-bd7d-44b5-902e-099da2ef429b\") " pod="openshift-ingress-canary/ingress-canary-s2zkr" Oct 30 17:58:22.206298 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.206284 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8s9\" (UniqueName: \"kubernetes.io/projected/57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a-kube-api-access-5g8s9\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j\" (UID: \"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" Oct 30 17:58:22.282107 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282072 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-kubelet\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.282107 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282105 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-bin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282127 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282157 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282206 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-kubelet\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282230 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282206 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-bin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282245 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282257 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-modprobe-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282212 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282301 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-host\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282328 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:22.282353 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282353 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysconfig\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.282378 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.782355246 +0000 UTC m=+1.831400160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282386 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-host\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282395 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-modprobe-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282402 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysconfig\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282402 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282435 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-crio-socket\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282448 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-etc-selinux\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282462 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f34929e-0b21-4743-af11-1360d39f162b-host\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.282453 4862 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282485 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-crio-socket\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282504 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-var-lib-kubelet\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282524 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f34929e-0b21-4743-af11-1360d39f162b-host\") pod \"node-ca-6wlg8\" (UID: \"4f34929e-0b21-4743-af11-1360d39f162b\") " pod="openshift-image-registry/node-ca-6wlg8" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.282539 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls podName:d8c35677-9ec2-4c68-938f-a9cfcc872f79 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.782521272 +0000 UTC m=+1.831566187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c2d9m" (UID: "d8c35677-9ec2-4c68-938f-a9cfcc872f79") : secret "samples-operator-tls" not found Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282558 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-var-lib-kubelet\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282561 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-sys-fs\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282597 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-dbus\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282608 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-sys-fs\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282618 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282643 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-ovn\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282669 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-system-cni-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282678 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-dbus\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282693 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282718 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-ovn\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.282722 4862 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282719 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-log-socket\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282754 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-log-socket\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282757 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-systemd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.282786 4862 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282790 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-multus\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.282801 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.782787718 +0000 UTC m=+1.831832608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : secret "router-metrics-certs-default" not found Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282721 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-system-cni-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282814 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-cni-multus\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.282818 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.782809218 +0000 UTC m=+1.831854143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : secret "metrics-daemon-secret" not found Oct 30 17:58:22.282829 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282850 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-node-log\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282863 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-systemd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282868 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282884 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-k8s-cni-cncf-io\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282894 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-node-log\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282902 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-kubernetes\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282921 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-k8s-cni-cncf-io\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282922 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-systemd\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282955 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-kubelet-config\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282973 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cnibin\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282979 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-systemd\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282956 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-kubernetes\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.282998 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-hostroot\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283016 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-cnibin\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283037 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-hostroot\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283026 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a8820c7-364d-4f01-90e2-ea9979976e47-kubelet-config\") pod \"global-pull-secret-syncer-dt49r\" (UID: \"5a8820c7-364d-4f01-90e2-ea9979976e47\") " pod="kube-system/global-pull-secret-syncer-dt49r" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283043 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283074 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283078 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-systemd-units\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283105 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-bin\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283132 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-system-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283149 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-systemd-units\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283159 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283176 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-bin\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283193 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-system-cni-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283194 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-host-slash\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283212 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/748c9d09-9db4-4e7d-9389-1b74c8e0e2c7-host-slash\") pod \"iptables-alerter-j8t4c\" (UID: \"748c9d09-9db4-4e7d-9389-1b74c8e0e2c7\") " pod="openshift-network-operator/iptables-alerter-j8t4c" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283219 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283231 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-conf-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283256 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-conf-dir\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283283 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283310 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-lib-modules\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283333 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283357 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-cnibin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283382 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-netns\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.283407 4862 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.283414 4862 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.283421 4862 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5c5549c-bgqzt: secret "image-registry-tls" not found Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283444 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.283458 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.783444866 +0000 UTC m=+1.832489770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283459 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-cnibin\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283442 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-multus-certs\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283407 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-multus-certs\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.283476 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls podName:93028940-79f3-45fa-be9b-6ddd8dab5633 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.783466751 +0000 UTC m=+1.832511644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls") pod "image-registry-74f5c5549c-bgqzt" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633") : secret "image-registry-tls" not found Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283420 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-lib-modules\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283470 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-run-netns\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283514 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-etc-kubernetes\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283540 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-etc-kubernetes\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283537 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-var-lib-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283567 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-var-lib-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283574 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-hosts-file\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283590 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283605 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283619 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-sys\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283633 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-registration-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283651 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-netns\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283679 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99e4d7d6-f980-4b03-8200-d6c5d8d3dc40-hosts-file\") pod \"node-resolver-rv2ks\" (UID: \"99e4d7d6-f980-4b03-8200-d6c5d8d3dc40\") " pod="openshift-dns/node-resolver-rv2ks" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283687 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56c4dbcb-db0a-4e7d-a633-3ff9304037d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p4pv4\" (UID: \"56c4dbcb-db0a-4e7d-a633-3ff9304037d4\") " pod="openshift-multus/multus-additional-cni-plugins-p4pv4" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283687 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-run-netns\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283703 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-registration-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283709 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-sys\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283734 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-run\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283765 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-kubelet\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.283772 4862 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283786 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-run\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283791 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-netd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.283806 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls podName:529241e2-eb4e-4412-ad1f-37dee4aa9b65 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.783793635 +0000 UTC m=+1.832838529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls") pod "dns-default-r94b8" (UID: "529241e2-eb4e-4412-ad1f-37dee4aa9b65") : secret "dns-default-metrics-tls" not found Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283824 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-cni-netd\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283852 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-host-var-lib-kubelet\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283860 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283877 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283895 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-conf\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283909 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.283928 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283958 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.283983 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.284007 4862 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284007 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-socket-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.284047 4862 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.284065 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.784047987 +0000 UTC m=+1.833092892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284051 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-conf\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.284069 4862 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55dcc4fcc6-4mw7c: secret "image-registry-tls" not found Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284045 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-etc-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284008 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66e91a23-8415-48a9-924c-64f752b59272-etc-sysctl-d\") pod \"tuned-zb8gk\" (UID: \"66e91a23-8415-48a9-924c-64f752b59272\") " pod="openshift-cluster-node-tuning-operator/tuned-zb8gk" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284095 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-etc-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.284107 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls podName:056042ce-b312-4d54-9f83-15cd37aa97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.784096394 +0000 UTC m=+1.833141342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls") pod "image-registry-55dcc4fcc6-4mw7c" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce") : secret "image-registry-tls" not found Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284063 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-run-openvswitch\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284106 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-socket-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.284130 4862 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284150 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-slash\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284179 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-socket-dir-parent\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.284193 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls podName:874e6413-424d-4443-80fc-f9e4d0a63b6c nodeName:}" failed. No retries permitted until 2026-10-30 17:58:22.784181948 +0000 UTC m=+1.833226838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8g5tv" (UID: "874e6413-424d-4443-80fc-f9e4d0a63b6c") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284193 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d79402ab-1f1f-458b-b7c8-3c8209e69efb-host-slash\") pod \"ovnkube-node-dx2gw\" (UID: \"d79402ab-1f1f-458b-b7c8-3c8209e69efb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284221 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-device-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284231 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab36263f-c154-4f9f-9a8f-10fc38405e52-multus-socket-dir-parent\") pod \"multus-ch8b8\" (UID: \"ab36263f-c154-4f9f-9a8f-10fc38405e52\") " pod="openshift-multus/multus-ch8b8" Oct 30 17:58:22.286072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.284309 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c598714d-0188-42bd-80e6-d9873dc2ae20-device-dir\") pod \"aws-ebs-csi-driver-node-v5psx\" (UID: \"c598714d-0188-42bd-80e6-d9873dc2ae20\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v5psx" Oct 30 17:58:22.374121 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.374087 4862 scope.go:122] "RemoveContainer" containerID="d3ca2364d68fe0246dca3420e6db36a2937baca9ad9ef1d17b53de359919a379" Oct 30 17:58:22.437832 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.437809 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:22.788869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.788784 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:22.788869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.788820 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:22.788869 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.788842 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.788937 4862 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.788939 4862 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.788980 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls podName:874e6413-424d-4443-80fc-f9e4d0a63b6c nodeName:}" failed. No retries permitted until 2026-10-30 17:58:23.788968655 +0000 UTC m=+2.838013565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8g5tv" (UID: "874e6413-424d-4443-80fc-f9e4d0a63b6c") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.788981 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789004 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:23.788988144 +0000 UTC m=+2.838033035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.788942 4862 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789038 4862 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55dcc4fcc6-4mw7c: secret "image-registry-tls" not found Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.789049 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789067 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:23.789056137 +0000 UTC m=+2.838101043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:22.789104 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789087 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls podName:056042ce-b312-4d54-9f83-15cd37aa97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:23.789076429 +0000 UTC m=+2.838121333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls") pod "image-registry-55dcc4fcc6-4mw7c" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce") : secret "image-registry-tls" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789117 4862 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.789123 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789161 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls podName:d8c35677-9ec2-4c68-938f-a9cfcc872f79 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:23.789148945 +0000 UTC m=+2.838193835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-c2d9m" (UID: "d8c35677-9ec2-4c68-938f-a9cfcc872f79") : secret "samples-operator-tls" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.789183 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.789254 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.789309 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.789344 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789408 4862 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789412 4862 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789428 4862 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5c5549c-bgqzt: secret "image-registry-tls" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789438 4862 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789449 4862 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.789460 4862 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789469 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:23.789438599 +0000 UTC m=+2.838483493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : secret "metrics-daemon-secret" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789553 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls podName:93028940-79f3-45fa-be9b-6ddd8dab5633 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:23.789530238 +0000 UTC m=+2.838575128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls") pod "image-registry-74f5c5549c-bgqzt" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633") : secret "image-registry-tls" not found Oct 30 17:58:22.789579 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789578 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:23.789566687 +0000 UTC m=+2.838611608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:22.790187 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:22.789600 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls podName:529241e2-eb4e-4412-ad1f-37dee4aa9b65 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:23.789590207 +0000 UTC m=+2.838635100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls") pod "dns-default-r94b8" (UID: "529241e2-eb4e-4412-ad1f-37dee4aa9b65") : secret "dns-default-metrics-tls" not found Oct 30 17:58:22.791982 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:22.791959 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b5e199e-2896-450f-9e73-838fb7e12e48-metrics-certs\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:23.108713 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.108686 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 17:58:23.109148 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.109121 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/1.log" Oct 30 17:58:23.109185 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.109149 4862 generic.go:356] "Generic (PLEG): container finished" podID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" containerID="400719f155a214730ff9164759f2fb4abf70eed25cc02aa7c13e3158b935c684" exitCode=255 Oct 30 17:58:23.109298 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.109254 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" event={"ID":"21caa3cf-ca4f-424a-9f40-6f91a7b176c3","Type":"ContainerDied","Data":"400719f155a214730ff9164759f2fb4abf70eed25cc02aa7c13e3158b935c684"} Oct 30 17:58:23.109339 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.109329 4862 scope.go:122] "RemoveContainer" containerID="d3ca2364d68fe0246dca3420e6db36a2937baca9ad9ef1d17b53de359919a379" Oct 30 17:58:23.110110 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.110088 4862 scope.go:122] "RemoveContainer" containerID="400719f155a214730ff9164759f2fb4abf70eed25cc02aa7c13e3158b935c684" Oct 30 17:58:23.110326 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.110299 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gxwj6_openshift-console-operator(21caa3cf-ca4f-424a-9f40-6f91a7b176c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" podUID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" Oct 30 17:58:23.225594 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.225560 4862 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vz7gh_7f619d36-116d-4434-bf1e-1aecb4d7fefe/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:23.440884 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.440805 4862 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:23.748697 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.748597 4862 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vz7gh_7f619d36-116d-4434-bf1e-1aecb4d7fefe/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:23.799775 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.799740 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:23.799872 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.799785 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:23.799872 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.799820 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:23.799872 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.799861 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:23.800011 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.799893 4862 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:23.800011 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.799895 4862 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:23.800011 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.799907 4862 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5c5549c-bgqzt: secret "image-registry-tls" not found Oct 30 17:58:23.800011 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.799897 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:23.800011 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.799961 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.799944666 +0000 UTC m=+4.848989572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:23.800011 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.799973 4862 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 17:58:23.800011 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.799983 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls podName:93028940-79f3-45fa-be9b-6ddd8dab5633 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.799974865 +0000 UTC m=+4.849019756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls") pod "image-registry-74f5c5549c-bgqzt" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633") : secret "image-registry-tls" not found Oct 30 17:58:23.800011 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.799994 4862 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:23.800011 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800003 4862 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800022 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls podName:529241e2-eb4e-4412-ad1f-37dee4aa9b65 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.800010143 +0000 UTC m=+4.849055035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls") pod "dns-default-r94b8" (UID: "529241e2-eb4e-4412-ad1f-37dee4aa9b65") : secret "dns-default-metrics-tls" not found Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.800021 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800057 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.800042301 +0000 UTC m=+4.849087208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800007 4862 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55dcc4fcc6-4mw7c: secret "image-registry-tls" not found Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800079 4862 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.800096 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800115 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls podName:874e6413-424d-4443-80fc-f9e4d0a63b6c nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.800105824 +0000 UTC m=+4.849150733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8g5tv" (UID: "874e6413-424d-4443-80fc-f9e4d0a63b6c") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800137 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls podName:056042ce-b312-4d54-9f83-15cd37aa97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.800130134 +0000 UTC m=+4.849175026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls") pod "image-registry-55dcc4fcc6-4mw7c" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce") : secret "image-registry-tls" not found Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.800158 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800167 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.800155728 +0000 UTC m=+4.849200624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.800214 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800289 4862 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:23.800461 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:23.800327 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:25.800316884 +0000 UTC m=+4.849361775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : secret "metrics-daemon-secret" not found Oct 30 17:58:23.802595 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.802569 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d8c35677-9ec2-4c68-938f-a9cfcc872f79-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-c2d9m\" (UID: \"d8c35677-9ec2-4c68-938f-a9cfcc872f79\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:23.875089 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:23.875057 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" Oct 30 17:58:24.114396 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:24.114362 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 17:58:24.114874 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:24.114754 4862 scope.go:122] "RemoveContainer" containerID="400719f155a214730ff9164759f2fb4abf70eed25cc02aa7c13e3158b935c684" Oct 30 17:58:24.114979 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:24.114956 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gxwj6_openshift-console-operator(21caa3cf-ca4f-424a-9f40-6f91a7b176c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" podUID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" Oct 30 17:58:24.169922 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:24.169889 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m"] Oct 30 17:58:24.219486 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:24.219470 4862 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:58:24.452074 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:24.451993 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rv2ks_99e4d7d6-f980-4b03-8200-d6c5d8d3dc40/dns-node-resolver/0.log" Oct 30 17:58:24.670357 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:24.670325 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:25.119612 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.119572 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" event={"ID":"d8c35677-9ec2-4c68-938f-a9cfcc872f79","Type":"ContainerStarted","Data":"62ca5626e4d535fce9fdd367c0a1235f353b1546f3c4b51bf869971453b7999f"} Oct 30 17:58:25.120011 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.119851 4862 scope.go:122] "RemoveContainer" containerID="400719f155a214730ff9164759f2fb4abf70eed25cc02aa7c13e3158b935c684" Oct 30 17:58:25.120084 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.120060 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gxwj6_openshift-console-operator(21caa3cf-ca4f-424a-9f40-6f91a7b176c3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" podUID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" Oct 30 17:58:25.249072 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.249041 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wlg8_4f34929e-0b21-4743-af11-1360d39f162b/node-ca/0.log" Oct 30 17:58:25.821366 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.821321 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:25.821541 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.821410 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:25.821541 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.821441 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:25.821541 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.821474 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:25.821541 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821490 4862 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:25.821541 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.821517 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:25.821781 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.821546 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:25.821781 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821570 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:29.82154763 +0000 UTC m=+8.870592610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : secret "metrics-daemon-secret" not found Oct 30 17:58:25.821781 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.821612 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:25.821781 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821650 4862 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:25.821781 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821665 4862 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55dcc4fcc6-4mw7c: secret "image-registry-tls" not found Oct 30 17:58:25.821781 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821714 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls podName:056042ce-b312-4d54-9f83-15cd37aa97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:29.821698441 +0000 UTC m=+8.870743337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls") pod "image-registry-55dcc4fcc6-4mw7c" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce") : secret "image-registry-tls" not found Oct 30 17:58:25.821781 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821731 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:29.821721456 +0000 UTC m=+8.870766356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:25.822051 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821789 4862 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 17:58:25.822051 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821798 4862 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74f5c5549c-bgqzt: secret "image-registry-tls" not found Oct 30 17:58:25.822051 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821828 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls podName:93028940-79f3-45fa-be9b-6ddd8dab5633 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:29.821818041 +0000 UTC m=+8.870862936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls") pod "image-registry-74f5c5549c-bgqzt" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633") : secret "image-registry-tls" not found Oct 30 17:58:25.822051 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821887 4862 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:25.822051 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.821918 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:29.821907531 +0000 UTC m=+8.870952424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:25.822307 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.822084 4862 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:25.822307 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.822125 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls podName:874e6413-424d-4443-80fc-f9e4d0a63b6c nodeName:}" failed. No retries permitted until 2026-10-30 17:58:29.822111076 +0000 UTC m=+8.871155982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8g5tv" (UID: "874e6413-424d-4443-80fc-f9e4d0a63b6c") : secret "cluster-monitoring-operator-tls" not found Oct 30 17:58:25.822307 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.822187 4862 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:25.822307 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:25.822220 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:29.82220867 +0000 UTC m=+8.871253575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:25.822307 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.821661 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:25.824673 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.824648 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/529241e2-eb4e-4412-ad1f-37dee4aa9b65-metrics-tls\") pod \"dns-default-r94b8\" (UID: \"529241e2-eb4e-4412-ad1f-37dee4aa9b65\") " pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:25.983496 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.983473 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:25.995730 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.995707 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:25.995959 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:25.995945 4862 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:26.124527 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:26.124408 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" event={"ID":"d8c35677-9ec2-4c68-938f-a9cfcc872f79","Type":"ContainerStarted","Data":"89293438dc12ad9e4df9d197b618ed6fb2e268cdcec86fdd5b89a168fadcdf09"} Oct 30 17:58:26.124527 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:26.124492 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-c2d9m" event={"ID":"d8c35677-9ec2-4c68-938f-a9cfcc872f79","Type":"ContainerStarted","Data":"412637c36c3e4b590930c68bfd0a3f439bd0285edb6e18dd6fd1acf58e892969"} Oct 30 17:58:26.279146 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:26.279116 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r94b8"] Oct 30 17:58:26.283840 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:26.283807 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529241e2_eb4e_4412_ad1f_37dee4aa9b65.slice/crio-54bf2afc732db7bd0cf04277ecb712b04891be54f117af65c59ec3aa6bac7db9 WatchSource:0}: Error finding container 54bf2afc732db7bd0cf04277ecb712b04891be54f117af65c59ec3aa6bac7db9: Status 404 returned error can't find the container with id 54bf2afc732db7bd0cf04277ecb712b04891be54f117af65c59ec3aa6bac7db9 Oct 30 17:58:26.448833 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:26.448771 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s2zkr_7caaae72-bd7d-44b5-902e-099da2ef429b/serve-healthcheck-canary/0.log" Oct 30 17:58:26.616513 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:26.616487 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:58:27.050835 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:27.050804 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j_57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a/kube-storage-version-migrator-operator/0.log" Oct 30 17:58:27.129219 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:27.129184 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r94b8" event={"ID":"529241e2-eb4e-4412-ad1f-37dee4aa9b65","Type":"ContainerStarted","Data":"54bf2afc732db7bd0cf04277ecb712b04891be54f117af65c59ec3aa6bac7db9"} Oct 30 17:58:28.137721 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:28.137690 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r94b8" event={"ID":"529241e2-eb4e-4412-ad1f-37dee4aa9b65","Type":"ContainerStarted","Data":"af963e2fa35ee93cad9ec9e9011e706dea653a7f2caae86c55954206a69f0059"} Oct 30 17:58:28.138077 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:28.137759 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r94b8" event={"ID":"529241e2-eb4e-4412-ad1f-37dee4aa9b65","Type":"ContainerStarted","Data":"5665e72384b9e21112261625bd36626fa10bf12d592fe76284593088c8c5432d"} Oct 30 17:58:28.138077 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:28.137842 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:29.858673 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.858640 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.858685 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.858705 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:29.858793 4862 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:29.858801 4862 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.858833 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:29.858841 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls podName:cc9f0fe3-754d-4da0-becb-f5484b5ab9dd nodeName:}" failed. No retries permitted until 2026-10-30 17:58:37.858828842 +0000 UTC m=+16.907873749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f2zht" (UID: "cc9f0fe3-754d-4da0-becb-f5484b5ab9dd") : secret "insights-runtime-extractor-tls" not found Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:29.858873 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs podName:1bc68151-0620-48d2-a803-e7cb588c5b76 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:37.858859716 +0000 UTC m=+16.907904607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs") pod "network-metrics-daemon-hk9ss" (UID: "1bc68151-0620-48d2-a803-e7cb588c5b76") : secret "metrics-daemon-secret" not found Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.858895 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:29.858910 4862 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.858915 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.858942 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:29.858980 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert podName:62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce nodeName:}" failed. No retries permitted until 2026-10-30 17:58:37.858969977 +0000 UTC m=+16.908014871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mk86v" (UID: "62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce") : secret "networking-console-plugin-cert" not found Oct 30 17:58:29.859160 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:29.859079 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle podName:5b5e199e-2896-450f-9e73-838fb7e12e48 nodeName:}" failed. No retries permitted until 2026-10-30 17:58:37.859061331 +0000 UTC m=+16.908106235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle") pod "router-default-ff9667845-fjrpd" (UID: "5b5e199e-2896-450f-9e73-838fb7e12e48") : configmap references non-existent config key: service-ca.crt Oct 30 17:58:29.861291 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.861253 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"image-registry-74f5c5549c-bgqzt\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:29.861391 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.861374 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/874e6413-424d-4443-80fc-f9e4d0a63b6c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8g5tv\" (UID: \"874e6413-424d-4443-80fc-f9e4d0a63b6c\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:29.861446 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.861374 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"image-registry-55dcc4fcc6-4mw7c\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:29.873159 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.873140 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" Oct 30 17:58:29.873327 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.873310 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:29.882361 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:29.882339 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:30.186931 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:30.186870 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv"] Oct 30 17:58:30.197392 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:30.197367 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874e6413_424d_4443_80fc_f9e4d0a63b6c.slice/crio-400d707a02fd4a5e5c556838fa760475136ecf1ff245080e6c0e071d2dfa8688 WatchSource:0}: Error finding container 400d707a02fd4a5e5c556838fa760475136ecf1ff245080e6c0e071d2dfa8688: Status 404 returned error can't find the container with id 400d707a02fd4a5e5c556838fa760475136ecf1ff245080e6c0e071d2dfa8688 Oct 30 17:58:30.197712 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:30.197693 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55dcc4fcc6-4mw7c"] Oct 30 17:58:30.197879 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:30.197863 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056042ce_b312_4d54_9f83_15cd37aa97ce.slice/crio-30b3a52c44bcd6db34902550e6d2178fe6c2f8dc1f9491d69d29e580ced8a638 WatchSource:0}: Error finding container 30b3a52c44bcd6db34902550e6d2178fe6c2f8dc1f9491d69d29e580ced8a638: Status 404 returned error can't find the container with id 30b3a52c44bcd6db34902550e6d2178fe6c2f8dc1f9491d69d29e580ced8a638 Oct 30 17:58:30.204527 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:30.204505 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74f5c5549c-bgqzt"] Oct 30 17:58:30.209164 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:30.209143 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93028940_79f3_45fa_be9b_6ddd8dab5633.slice/crio-394b5fb7154d0dde0a0a2eee4be66c0d40f2f8f822adf7449ac4f208b079c681 WatchSource:0}: Error finding container 394b5fb7154d0dde0a0a2eee4be66c0d40f2f8f822adf7449ac4f208b079c681: Status 404 returned error can't find the container with id 394b5fb7154d0dde0a0a2eee4be66c0d40f2f8f822adf7449ac4f208b079c681 Oct 30 17:58:30.553402 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:30.553368 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx2gw" Oct 30 17:58:31.148906 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:31.148873 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" event={"ID":"056042ce-b312-4d54-9f83-15cd37aa97ce","Type":"ContainerStarted","Data":"be32926f9a97cecb31c91d4c1682d75c16f73367f0814bd7b94d8af821d6adb1"} Oct 30 17:58:31.149408 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:31.148927 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" event={"ID":"056042ce-b312-4d54-9f83-15cd37aa97ce","Type":"ContainerStarted","Data":"30b3a52c44bcd6db34902550e6d2178fe6c2f8dc1f9491d69d29e580ced8a638"} Oct 30 17:58:31.150718 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:31.150677 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" event={"ID":"93028940-79f3-45fa-be9b-6ddd8dab5633","Type":"ContainerStarted","Data":"92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670"} Oct 30 17:58:31.150838 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:31.150726 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" event={"ID":"93028940-79f3-45fa-be9b-6ddd8dab5633","Type":"ContainerStarted","Data":"394b5fb7154d0dde0a0a2eee4be66c0d40f2f8f822adf7449ac4f208b079c681"} Oct 30 17:58:31.152726 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:31.152698 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" event={"ID":"874e6413-424d-4443-80fc-f9e4d0a63b6c","Type":"ContainerStarted","Data":"400d707a02fd4a5e5c556838fa760475136ecf1ff245080e6c0e071d2dfa8688"} Oct 30 17:58:32.157195 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:32.157153 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8g5tv" event={"ID":"874e6413-424d-4443-80fc-f9e4d0a63b6c","Type":"ContainerStarted","Data":"8c8758e971f22d10182c464f4838444213686f73aef49ea8b65ebc408b96e59f"} Oct 30 17:58:32.157687 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:32.157199 4862 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:32.157687 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:32.157217 4862 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:58:35.055092 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:35.055062 4862 scope.go:122] "RemoveContainer" containerID="400719f155a214730ff9164759f2fb4abf70eed25cc02aa7c13e3158b935c684" Oct 30 17:58:35.169413 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:35.169378 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 17:58:35.169549 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:35.169433 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" event={"ID":"21caa3cf-ca4f-424a-9f40-6f91a7b176c3","Type":"ContainerStarted","Data":"8e99cb62f3ff5831ecc5ff8bf9b92948777eccee5abb1bfa02dae8ae724d3c0a"} Oct 30 17:58:35.169702 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:35.169679 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:35.170406 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:35.170375 4862 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-gxwj6 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.19:8443/readyz\": dial tcp 10.133.0.19:8443: connect: connection refused" start-of-body= Oct 30 17:58:35.170489 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:35.170413 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" podUID="21caa3cf-ca4f-424a-9f40-6f91a7b176c3" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.19:8443/readyz\": dial tcp 10.133.0.19:8443: connect: connection refused" Oct 30 17:58:35.601348 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:35.601321 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:36.177061 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:36.177032 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gxwj6" Oct 30 17:58:37.434945 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.434918 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:37.927336 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.927306 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:37.927503 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.927348 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:37.927503 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.927385 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:37.927503 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.927416 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:37.928016 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.927985 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b5e199e-2896-450f-9e73-838fb7e12e48-service-ca-bundle\") pod \"router-default-ff9667845-fjrpd\" (UID: \"5b5e199e-2896-450f-9e73-838fb7e12e48\") " pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:37.930012 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.929988 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc9f0fe3-754d-4da0-becb-f5484b5ab9dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f2zht\" (UID: \"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd\") " pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:37.930106 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.929988 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bc68151-0620-48d2-a803-e7cb588c5b76-metrics-certs\") pod \"network-metrics-daemon-hk9ss\" (UID: \"1bc68151-0620-48d2-a803-e7cb588c5b76\") " pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:37.930106 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.930095 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mk86v\" (UID: \"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:37.973403 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.973373 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:37.976479 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.976462 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" Oct 30 17:58:37.981354 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.981339 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk9ss" Oct 30 17:58:37.984146 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:37.984126 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f2zht" Oct 30 17:58:38.142744 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:38.142719 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r94b8" Oct 30 17:58:38.262059 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:38.261995 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74f5c5549c-bgqzt"] Oct 30 17:58:38.323767 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:38.323734 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hk9ss"] Oct 30 17:58:38.328553 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:38.326040 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mk86v"] Oct 30 17:58:38.328553 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:38.326799 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f2zht"] Oct 30 17:58:38.330279 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:38.330241 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc68151_0620_48d2_a803_e7cb588c5b76.slice/crio-1309220b380b9b13ec7923556572c586ddfa58093fe34dbc96dc05621e972ccd WatchSource:0}: Error finding container 1309220b380b9b13ec7923556572c586ddfa58093fe34dbc96dc05621e972ccd: Status 404 returned error can't find the container with id 1309220b380b9b13ec7923556572c586ddfa58093fe34dbc96dc05621e972ccd Oct 30 17:58:38.331054 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:38.331026 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fc3e8a_9a9e_4b43_9cc7_dee3cdcb97ce.slice/crio-8faa3048d2c84f33302fe0b25496d262e658e3eb33f21b169d300d398e2b1ca5 WatchSource:0}: Error finding container 8faa3048d2c84f33302fe0b25496d262e658e3eb33f21b169d300d398e2b1ca5: Status 404 returned error can't find the container with id 8faa3048d2c84f33302fe0b25496d262e658e3eb33f21b169d300d398e2b1ca5 Oct 30 17:58:38.331926 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:38.331903 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9f0fe3_754d_4da0_becb_f5484b5ab9dd.slice/crio-1dbfcd76236d9cf894ba6dd065ebdee4ac2a3625742129817c6b0465b0b2de90 WatchSource:0}: Error finding container 1dbfcd76236d9cf894ba6dd065ebdee4ac2a3625742129817c6b0465b0b2de90: Status 404 returned error can't find the container with id 1dbfcd76236d9cf894ba6dd065ebdee4ac2a3625742129817c6b0465b0b2de90 Oct 30 17:58:38.337018 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:38.336541 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ff9667845-fjrpd"] Oct 30 17:58:38.340132 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:38.340107 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5e199e_2896_450f_9e73_838fb7e12e48.slice/crio-fa2c5612abd2ac5b98f496b4fecc8c6918c35cfb16f6e4709ff31d0b8c02a9ae WatchSource:0}: Error finding container fa2c5612abd2ac5b98f496b4fecc8c6918c35cfb16f6e4709ff31d0b8c02a9ae: Status 404 returned error can't find the container with id fa2c5612abd2ac5b98f496b4fecc8c6918c35cfb16f6e4709ff31d0b8c02a9ae Oct 30 17:58:38.890163 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:38.889546 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vz7gh" Oct 30 17:58:39.186179 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:39.186100 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk9ss" event={"ID":"1bc68151-0620-48d2-a803-e7cb588c5b76","Type":"ContainerStarted","Data":"1309220b380b9b13ec7923556572c586ddfa58093fe34dbc96dc05621e972ccd"} Oct 30 17:58:39.188519 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:39.188493 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ff9667845-fjrpd" event={"ID":"5b5e199e-2896-450f-9e73-838fb7e12e48","Type":"ContainerStarted","Data":"73e51352725ae1f0dad226ff4d5d7982bed2f596e2cee630f3db79c1ea3d0ee8"} Oct 30 17:58:39.188661 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:39.188532 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ff9667845-fjrpd" event={"ID":"5b5e199e-2896-450f-9e73-838fb7e12e48","Type":"ContainerStarted","Data":"fa2c5612abd2ac5b98f496b4fecc8c6918c35cfb16f6e4709ff31d0b8c02a9ae"} Oct 30 17:58:39.191532 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:39.191506 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2zht" event={"ID":"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd","Type":"ContainerStarted","Data":"621071a6e0c6aded905c970da3fea631d1357c9a313a0200b6bf81d13c8a13a4"} Oct 30 17:58:39.191644 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:39.191556 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2zht" event={"ID":"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd","Type":"ContainerStarted","Data":"d4568078c71496213d8eece429e7a502ddc104a460fe6979a9003cf1088cdd2f"} Oct 30 17:58:39.191644 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:39.191585 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2zht" event={"ID":"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd","Type":"ContainerStarted","Data":"1dbfcd76236d9cf894ba6dd065ebdee4ac2a3625742129817c6b0465b0b2de90"} Oct 30 17:58:39.193128 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:39.193096 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" event={"ID":"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce","Type":"ContainerStarted","Data":"8faa3048d2c84f33302fe0b25496d262e658e3eb33f21b169d300d398e2b1ca5"} Oct 30 17:58:39.973805 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:39.973773 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:39.976873 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:39.976849 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:40.198666 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:40.198626 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-mk86v" event={"ID":"62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce","Type":"ContainerStarted","Data":"0f3e0746856fe42a37d23ca62dfe54a401206638e1791f99e24b8796b2ac0607"} Oct 30 17:58:40.200664 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:40.200624 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk9ss" event={"ID":"1bc68151-0620-48d2-a803-e7cb588c5b76","Type":"ContainerStarted","Data":"d17aab18772600afc52b3c81b8c0023e6fa44b0a5c2a74673e2ecb5d462b5096"} Oct 30 17:58:40.200802 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:40.200691 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk9ss" event={"ID":"1bc68151-0620-48d2-a803-e7cb588c5b76","Type":"ContainerStarted","Data":"471a24d8dc2f88f45ee37994fed6630c4820f98efcc6a2052bf21d047a02ba71"} Oct 30 17:58:40.201076 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:40.201054 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:40.202367 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:40.202350 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-ff9667845-fjrpd" Oct 30 17:58:41.206140 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:41.206096 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f2zht" event={"ID":"cc9f0fe3-754d-4da0-becb-f5484b5ab9dd","Type":"ContainerStarted","Data":"e8accc0dbe1cc79df44095dea76e4c059cc098caf2b308ebd9c7e9fe59c5d403"} Oct 30 17:58:45.606358 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.606322 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:58:45.607296 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.607277 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7wmpk"] Oct 30 17:58:45.642894 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.642867 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.668017 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.667994 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:58:45.668017 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.668027 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rlt9d\"" Oct 30 17:58:45.668178 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.668059 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:58:45.674422 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.674404 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:58:45.674541 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.674527 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:58:45.684004 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.683978 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-metrics-client-ca\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.684120 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.684071 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.684252 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.684225 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-sys\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.684381 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.684277 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.684381 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.684361 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-wtmp\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.684566 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.684548 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-root\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.684674 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.684583 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-tls\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.684866 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.684818 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99gd\" (UniqueName: \"kubernetes.io/projected/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-kube-api-access-j99gd\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.684910 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.684890 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-textfile\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.785851 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.785823 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-metrics-client-ca\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.785997 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.785856 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.785997 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.785884 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-sys\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.785997 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.785901 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.785997 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.785919 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-wtmp\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.785997 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.785970 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-root\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.785997 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.785971 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-sys\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.785997 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.785998 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-tls\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.786335 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.786020 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99gd\" (UniqueName: \"kubernetes.io/projected/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-kube-api-access-j99gd\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.786335 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.786045 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-textfile\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.786335 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.786059 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-root\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.786335 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.786110 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-wtmp\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.786525 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.786482 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-metrics-client-ca\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.786525 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.786484 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-textfile\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.786525 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.786520 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.788349 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.788326 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.788524 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.788507 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-node-exporter-tls\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.804253 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.804230 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99gd\" (UniqueName: \"kubernetes.io/projected/f2f8a427-9fa5-43c2-b914-801d52dbc3cf-kube-api-access-j99gd\") pod \"node-exporter-7wmpk\" (UID: \"f2f8a427-9fa5-43c2-b914-801d52dbc3cf\") " pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:45.992896 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:45.992823 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wmpk" Oct 30 17:58:46.027360 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:58:46.027317 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f8a427_9fa5_43c2_b914_801d52dbc3cf.slice/crio-12cb383af83ed93b2aca27f7925a3255159c8327fdd7308e8a39251ca54c0666 WatchSource:0}: Error finding container 12cb383af83ed93b2aca27f7925a3255159c8327fdd7308e8a39251ca54c0666: Status 404 returned error can't find the container with id 12cb383af83ed93b2aca27f7925a3255159c8327fdd7308e8a39251ca54c0666 Oct 30 17:58:46.226358 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:46.226324 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wmpk" event={"ID":"f2f8a427-9fa5-43c2-b914-801d52dbc3cf","Type":"ContainerStarted","Data":"12cb383af83ed93b2aca27f7925a3255159c8327fdd7308e8a39251ca54c0666"} Oct 30 17:58:48.239301 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:48.237923 4862 generic.go:356] "Generic (PLEG): container finished" podID="f2f8a427-9fa5-43c2-b914-801d52dbc3cf" containerID="b822f67323a8d46e3ffc41bd58064b2d6c0914f77b6ffeea4256b54c03948c92" exitCode=0 Oct 30 17:58:48.239301 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:48.237995 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wmpk" event={"ID":"f2f8a427-9fa5-43c2-b914-801d52dbc3cf","Type":"ContainerDied","Data":"b822f67323a8d46e3ffc41bd58064b2d6c0914f77b6ffeea4256b54c03948c92"} Oct 30 17:58:48.266142 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:48.266120 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:58:49.243752 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:49.243721 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wmpk" event={"ID":"f2f8a427-9fa5-43c2-b914-801d52dbc3cf","Type":"ContainerStarted","Data":"2fecc5667949804ab3165505ba9fdf1acf971f308502acff3c92bd218a0b02c7"} Oct 30 17:58:49.244126 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:49.243772 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wmpk" event={"ID":"f2f8a427-9fa5-43c2-b914-801d52dbc3cf","Type":"ContainerStarted","Data":"e418c02b037a3a21acec1f9a7b489cef08b2785fb899ea9f926ff38b1c655628"} Oct 30 17:58:49.283535 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:49.283477 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7wmpk" podStartSLOduration=2.7702714630000003 podStartE2EDuration="4.28346611s" totalImagesPullingTime="1.418663687s" totalInitContainerRuntime="94.53096ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:49.282693453 +0000 UTC m=+28.331738379" watchObservedRunningTime="2026-10-30 17:58:49.28346611 +0000 UTC m=+28.332511019" Oct 30 17:58:49.868122 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:58:49.868080 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 17:58:56.619179 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:58:56.619145 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q74cr" Oct 30 17:59:00.237798 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:00.237769 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55dcc4fcc6-4mw7c"] Oct 30 17:59:03.282893 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.282781 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" podUID="93028940-79f3-45fa-be9b-6ddd8dab5633" containerName="registry" containerID="cri-o://92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670" gracePeriod=30 Oct 30 17:59:03.529720 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.529697 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:59:03.626716 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.626690 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration\") pod \"93028940-79f3-45fa-be9b-6ddd8dab5633\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " Oct 30 17:59:03.626911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.626732 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca\") pod \"93028940-79f3-45fa-be9b-6ddd8dab5633\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " Oct 30 17:59:03.626911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.626762 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp\") pod \"93028940-79f3-45fa-be9b-6ddd8dab5633\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " Oct 30 17:59:03.626911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.626787 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token\") pod \"93028940-79f3-45fa-be9b-6ddd8dab5633\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " Oct 30 17:59:03.626911 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.626855 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates\") pod \"93028940-79f3-45fa-be9b-6ddd8dab5633\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " Oct 30 17:59:03.627131 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.626980 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted\") pod \"93028940-79f3-45fa-be9b-6ddd8dab5633\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " Oct 30 17:59:03.627131 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.627037 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets\") pod \"93028940-79f3-45fa-be9b-6ddd8dab5633\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " Oct 30 17:59:03.627131 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.627101 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") pod \"93028940-79f3-45fa-be9b-6ddd8dab5633\" (UID: \"93028940-79f3-45fa-be9b-6ddd8dab5633\") " Oct 30 17:59:03.627320 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.627159 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca" pod "93028940-79f3-45fa-be9b-6ddd8dab5633" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:03.627384 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.627343 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates" pod "93028940-79f3-45fa-be9b-6ddd8dab5633" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:03.627516 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.627502 4862 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-trusted-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:03.627593 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.627523 4862 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-certificates\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:03.629838 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.629776 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration" pod "93028940-79f3-45fa-be9b-6ddd8dab5633" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:03.629962 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.629851 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp" pod "93028940-79f3-45fa-be9b-6ddd8dab5633" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633"). InnerVolumeSpecName "kube-api-access-cl9jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:03.629962 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.629892 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token" pod "93028940-79f3-45fa-be9b-6ddd8dab5633" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:03.630074 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.630021 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls" pod "93028940-79f3-45fa-be9b-6ddd8dab5633" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:03.630074 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.630025 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets" pod "93028940-79f3-45fa-be9b-6ddd8dab5633" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:03.636348 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.636255 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted" pod "93028940-79f3-45fa-be9b-6ddd8dab5633" (UID: "93028940-79f3-45fa-be9b-6ddd8dab5633"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:03.728549 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.728523 4862 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93028940-79f3-45fa-be9b-6ddd8dab5633-ca-trust-extracted\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:03.728549 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.728547 4862 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-installation-pull-secrets\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:03.728704 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.728558 4862 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-registry-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:03.728704 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.728570 4862 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93028940-79f3-45fa-be9b-6ddd8dab5633-image-registry-private-configuration\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:03.728704 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.728579 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl9jp\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-kube-api-access-cl9jp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:03.728704 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:03.728587 4862 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93028940-79f3-45fa-be9b-6ddd8dab5633-bound-sa-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:04.305188 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:04.305114 4862 generic.go:356] "Generic (PLEG): container finished" podID="93028940-79f3-45fa-be9b-6ddd8dab5633" containerID="92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670" exitCode=0 Oct 30 17:59:04.305188 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:04.305188 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" Oct 30 17:59:04.305659 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:04.305199 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" event={"ID":"93028940-79f3-45fa-be9b-6ddd8dab5633","Type":"ContainerDied","Data":"92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670"} Oct 30 17:59:04.305659 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:04.305290 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f5c5549c-bgqzt" event={"ID":"93028940-79f3-45fa-be9b-6ddd8dab5633","Type":"ContainerDied","Data":"394b5fb7154d0dde0a0a2eee4be66c0d40f2f8f822adf7449ac4f208b079c681"} Oct 30 17:59:04.305659 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:04.305312 4862 scope.go:122] "RemoveContainer" containerID="92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670" Oct 30 17:59:04.313680 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:04.313660 4862 scope.go:122] "RemoveContainer" containerID="92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670" Oct 30 17:59:04.313955 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:59:04.313935 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670\": container with ID starting with 92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670 not found: ID does not exist" containerID="92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670" Oct 30 17:59:04.313997 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:04.313963 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670"} err="failed to get container status \"92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670\": rpc error: code = NotFound desc = could not find container \"92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670\": container with ID starting with 92401991a3ea60318d464effb64ebc6b3eed242c0e3015472d85394113327670 not found: ID does not exist" Oct 30 17:59:04.362045 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:04.362017 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74f5c5549c-bgqzt"] Oct 30 17:59:04.372297 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:04.372254 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74f5c5549c-bgqzt"] Oct 30 17:59:05.058921 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:05.058892 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93028940-79f3-45fa-be9b-6ddd8dab5633" path="/var/lib/kubelet/pods/93028940-79f3-45fa-be9b-6ddd8dab5633/volumes" Oct 30 17:59:19.868825 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:59:19.868789 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 17:59:25.258900 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.258843 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" podUID="056042ce-b312-4d54-9f83-15cd37aa97ce" containerName="registry" containerID="cri-o://be32926f9a97cecb31c91d4c1682d75c16f73367f0814bd7b94d8af821d6adb1" gracePeriod=30 Oct 30 17:59:25.396313 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.396285 4862 generic.go:356] "Generic (PLEG): container finished" podID="056042ce-b312-4d54-9f83-15cd37aa97ce" containerID="be32926f9a97cecb31c91d4c1682d75c16f73367f0814bd7b94d8af821d6adb1" exitCode=0 Oct 30 17:59:25.396451 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.396375 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" event={"ID":"056042ce-b312-4d54-9f83-15cd37aa97ce","Type":"ContainerDied","Data":"be32926f9a97cecb31c91d4c1682d75c16f73367f0814bd7b94d8af821d6adb1"} Oct 30 17:59:25.499984 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.499959 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:59:25.603989 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.603956 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token\") pod \"056042ce-b312-4d54-9f83-15cd37aa97ce\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " Oct 30 17:59:25.604184 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.604024 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") pod \"056042ce-b312-4d54-9f83-15cd37aa97ce\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " Oct 30 17:59:25.604184 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.604093 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration\") pod \"056042ce-b312-4d54-9f83-15cd37aa97ce\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " Oct 30 17:59:25.604184 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.604123 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted\") pod \"056042ce-b312-4d54-9f83-15cd37aa97ce\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " Oct 30 17:59:25.604184 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.604159 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss\") pod \"056042ce-b312-4d54-9f83-15cd37aa97ce\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " Oct 30 17:59:25.604429 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.604192 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates\") pod \"056042ce-b312-4d54-9f83-15cd37aa97ce\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " Oct 30 17:59:25.604429 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.604241 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets\") pod \"056042ce-b312-4d54-9f83-15cd37aa97ce\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " Oct 30 17:59:25.604429 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.604294 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca\") pod \"056042ce-b312-4d54-9f83-15cd37aa97ce\" (UID: \"056042ce-b312-4d54-9f83-15cd37aa97ce\") " Oct 30 17:59:25.604847 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.604668 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates" pod "056042ce-b312-4d54-9f83-15cd37aa97ce" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:25.605158 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.605093 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca" pod "056042ce-b312-4d54-9f83-15cd37aa97ce" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:25.606656 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.606624 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls" pod "056042ce-b312-4d54-9f83-15cd37aa97ce" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:25.607043 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.607016 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token" pod "056042ce-b312-4d54-9f83-15cd37aa97ce" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:25.607184 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.607160 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss" pod "056042ce-b312-4d54-9f83-15cd37aa97ce" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce"). InnerVolumeSpecName "kube-api-access-bx4ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:25.607184 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.607171 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration" pod "056042ce-b312-4d54-9f83-15cd37aa97ce" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:25.607315 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.607290 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets" pod "056042ce-b312-4d54-9f83-15cd37aa97ce" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:25.610740 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.610718 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted" pod "056042ce-b312-4d54-9f83-15cd37aa97ce" (UID: "056042ce-b312-4d54-9f83-15cd37aa97ce"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:25.705785 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.705755 4862 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-bound-sa-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:25.705785 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.705780 4862 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:25.705785 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.705792 4862 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-image-registry-private-configuration\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:25.706002 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.705801 4862 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/056042ce-b312-4d54-9f83-15cd37aa97ce-ca-trust-extracted\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:25.706002 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.705811 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx4ss\" (UniqueName: \"kubernetes.io/projected/056042ce-b312-4d54-9f83-15cd37aa97ce-kube-api-access-bx4ss\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:25.706002 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.705819 4862 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-registry-certificates\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:25.706002 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.705828 4862 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/056042ce-b312-4d54-9f83-15cd37aa97ce-installation-pull-secrets\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:25.706002 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:25.705836 4862 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/056042ce-b312-4d54-9f83-15cd37aa97ce-trusted-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 17:59:26.401521 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:26.401493 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" Oct 30 17:59:26.401944 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:26.401495 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55dcc4fcc6-4mw7c" event={"ID":"056042ce-b312-4d54-9f83-15cd37aa97ce","Type":"ContainerDied","Data":"30b3a52c44bcd6db34902550e6d2178fe6c2f8dc1f9491d69d29e580ced8a638"} Oct 30 17:59:26.401944 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:26.401624 4862 scope.go:122] "RemoveContainer" containerID="be32926f9a97cecb31c91d4c1682d75c16f73367f0814bd7b94d8af821d6adb1" Oct 30 17:59:26.457200 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:26.457170 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55dcc4fcc6-4mw7c"] Oct 30 17:59:26.467793 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:26.467767 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55dcc4fcc6-4mw7c"] Oct 30 17:59:27.059539 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:27.059470 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056042ce-b312-4d54-9f83-15cd37aa97ce" path="/var/lib/kubelet/pods/056042ce-b312-4d54-9f83-15cd37aa97ce/volumes" Oct 30 17:59:27.406346 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:27.406250 4862 generic.go:356] "Generic (PLEG): container finished" podID="476cffce-abe2-409d-9b4a-792af41d8f1c" containerID="399000c07e501b9e15a6c6646863cd8d0e4b5e5b1f5a3b8037313e120f85bd20" exitCode=0 Oct 30 17:59:27.406346 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:27.406327 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zg24h" event={"ID":"476cffce-abe2-409d-9b4a-792af41d8f1c","Type":"ContainerDied","Data":"399000c07e501b9e15a6c6646863cd8d0e4b5e5b1f5a3b8037313e120f85bd20"} Oct 30 17:59:27.406777 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:27.406716 4862 scope.go:122] "RemoveContainer" containerID="399000c07e501b9e15a6c6646863cd8d0e4b5e5b1f5a3b8037313e120f85bd20" Oct 30 17:59:28.182888 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.182857 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fn4jd-xgpg5"] Oct 30 17:59:28.183166 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.183154 4862 cpu_manager.go:414] "containerMap: removing container" podUID="93028940-79f3-45fa-be9b-6ddd8dab5633" containerName="registry" Oct 30 17:59:28.183216 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.183167 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93028940-79f3-45fa-be9b-6ddd8dab5633" containerName="registry" Oct 30 17:59:28.183216 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.183186 4862 cpu_manager.go:414] "containerMap: removing container" podUID="056042ce-b312-4d54-9f83-15cd37aa97ce" containerName="registry" Oct 30 17:59:28.183216 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.183191 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056042ce-b312-4d54-9f83-15cd37aa97ce" containerName="registry" Oct 30 17:59:28.183354 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.183251 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="056042ce-b312-4d54-9f83-15cd37aa97ce" containerName="registry" Oct 30 17:59:28.183354 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.183275 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="93028940-79f3-45fa-be9b-6ddd8dab5633" containerName="registry" Oct 30 17:59:28.186482 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.186460 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.212332 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.212310 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fn4jd-xgpg5"] Oct 30 17:59:28.324760 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.324729 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-service-ca-bundle\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.324925 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.324773 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-trusted-ca-bundle\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.324925 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.324798 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/468f44ff-aa39-4909-9133-b09ef48b4279-kube-api-access-2hfc7\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.325000 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.324930 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/468f44ff-aa39-4909-9133-b09ef48b4279-archives-path\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.412630 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.412600 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zg24h" event={"ID":"476cffce-abe2-409d-9b4a-792af41d8f1c","Type":"ContainerStarted","Data":"50ab75535fde3fe0809f55d25e889fc5f78e0ff44989aa196666cd838a1d5e72"} Oct 30 17:59:28.425637 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.425614 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/468f44ff-aa39-4909-9133-b09ef48b4279-archives-path\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.425739 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.425644 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-service-ca-bundle\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.425739 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.425674 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-trusted-ca-bundle\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.425739 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.425696 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/468f44ff-aa39-4909-9133-b09ef48b4279-kube-api-access-2hfc7\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.426042 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.426021 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/468f44ff-aa39-4909-9133-b09ef48b4279-archives-path\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.426313 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.426292 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-service-ca-bundle\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.426497 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.426482 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-trusted-ca-bundle\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.443436 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.443386 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/468f44ff-aa39-4909-9133-b09ef48b4279-kube-api-access-2hfc7\") pod \"periodic-gathering-fn4jd-xgpg5\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.497130 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.497084 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 17:59:28.791360 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:28.791211 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fn4jd-xgpg5"] Oct 30 17:59:28.795383 ip-10-0-128-246 kubenswrapper[4862]: W1030 17:59:28.795360 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468f44ff_aa39_4909_9133_b09ef48b4279.slice/crio-b499a0d9b85e4bf52d826598aebc6d3f3e7c7629f29a208df81d8aff8cae753f WatchSource:0}: Error finding container b499a0d9b85e4bf52d826598aebc6d3f3e7c7629f29a208df81d8aff8cae753f: Status 404 returned error can't find the container with id b499a0d9b85e4bf52d826598aebc6d3f3e7c7629f29a208df81d8aff8cae753f Oct 30 17:59:29.418664 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:29.418625 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" event={"ID":"468f44ff-aa39-4909-9133-b09ef48b4279","Type":"ContainerStarted","Data":"baeb74b2bea2b68b872d50522288b5c5f56590fef7262b3d1564b21201ba15a2"} Oct 30 17:59:29.419066 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:29.418681 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" event={"ID":"468f44ff-aa39-4909-9133-b09ef48b4279","Type":"ContainerStarted","Data":"b499a0d9b85e4bf52d826598aebc6d3f3e7c7629f29a208df81d8aff8cae753f"} Oct 30 17:59:29.458309 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:29.458246 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" podStartSLOduration=1.458234679 podStartE2EDuration="1.458234679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:29.457601042 +0000 UTC m=+68.506645938" watchObservedRunningTime="2026-10-30 17:59:29.458234679 +0000 UTC m=+68.507279589" Oct 30 17:59:31.427130 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:31.427100 4862 generic.go:356] "Generic (PLEG): container finished" podID="b810f479-5f8c-4acc-89de-a41d288c8e2c" containerID="7c0bdfbad9a90587da858bb70fe70ae0e6fcf973890511a5dd43a249703298c5" exitCode=0 Oct 30 17:59:31.427497 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:31.427144 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" event={"ID":"b810f479-5f8c-4acc-89de-a41d288c8e2c","Type":"ContainerDied","Data":"7c0bdfbad9a90587da858bb70fe70ae0e6fcf973890511a5dd43a249703298c5"} Oct 30 17:59:31.427497 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:31.427442 4862 scope.go:122] "RemoveContainer" containerID="7c0bdfbad9a90587da858bb70fe70ae0e6fcf973890511a5dd43a249703298c5" Oct 30 17:59:32.432908 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:32.432860 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-wcpjz" event={"ID":"b810f479-5f8c-4acc-89de-a41d288c8e2c","Type":"ContainerStarted","Data":"5e8e12620098c5a0a18f35ce5ac301b64a2587372be2e0f02defe0e7b9e3c067"} Oct 30 17:59:47.490711 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:47.490677 4862 generic.go:356] "Generic (PLEG): container finished" podID="57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a" containerID="72bc74cc18464467437f46d113ba54c226ba003b356fdd4fb927eaa60ca187e4" exitCode=0 Oct 30 17:59:47.491074 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:47.490745 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" event={"ID":"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a","Type":"ContainerDied","Data":"72bc74cc18464467437f46d113ba54c226ba003b356fdd4fb927eaa60ca187e4"} Oct 30 17:59:47.491074 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:47.491042 4862 scope.go:122] "RemoveContainer" containerID="72bc74cc18464467437f46d113ba54c226ba003b356fdd4fb927eaa60ca187e4" Oct 30 17:59:48.496086 ip-10-0-128-246 kubenswrapper[4862]: I1030 17:59:48.496052 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fmq9j" event={"ID":"57507bb6-f2a5-4bbd-9c67-33dc3c81ee3a","Type":"ContainerStarted","Data":"2a646624893ed7ed2bf4612ea0ff5e29d88f427da477a36433be547583cb95b1"} Oct 30 17:59:49.867556 ip-10-0-128-246 kubenswrapper[4862]: E1030 17:59:49.867515 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:00:19.867564 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:00:19.867523 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:00:49.868466 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:00:49.868362 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:00:53.729017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:53.728988 4862 generic.go:356] "Generic (PLEG): container finished" podID="468f44ff-aa39-4909-9133-b09ef48b4279" containerID="baeb74b2bea2b68b872d50522288b5c5f56590fef7262b3d1564b21201ba15a2" exitCode=0 Oct 30 18:00:53.729408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:53.729047 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" event={"ID":"468f44ff-aa39-4909-9133-b09ef48b4279","Type":"ContainerDied","Data":"baeb74b2bea2b68b872d50522288b5c5f56590fef7262b3d1564b21201ba15a2"} Oct 30 18:00:54.860570 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.860547 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 18:00:54.868686 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.868665 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468f44ff-aa39-4909-9133-b09ef48b4279-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/468f44ff-aa39-4909-9133-b09ef48b4279-archives-path\") pod \"468f44ff-aa39-4909-9133-b09ef48b4279\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " Oct 30 18:00:54.868792 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.868735 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-trusted-ca-bundle\") pod \"468f44ff-aa39-4909-9133-b09ef48b4279\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " Oct 30 18:00:54.868858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.868808 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468f44ff-aa39-4909-9133-b09ef48b4279-kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/468f44ff-aa39-4909-9133-b09ef48b4279-kube-api-access-2hfc7\") pod \"468f44ff-aa39-4909-9133-b09ef48b4279\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " Oct 30 18:00:54.868911 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.868854 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-service-ca-bundle\") pod \"468f44ff-aa39-4909-9133-b09ef48b4279\" (UID: \"468f44ff-aa39-4909-9133-b09ef48b4279\") " Oct 30 18:00:54.869000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.868974 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468f44ff-aa39-4909-9133-b09ef48b4279-archives-path" pod "468f44ff-aa39-4909-9133-b09ef48b4279" (UID: "468f44ff-aa39-4909-9133-b09ef48b4279"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:54.869126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.869105 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-trusted-ca-bundle" pod "468f44ff-aa39-4909-9133-b09ef48b4279" (UID: "468f44ff-aa39-4909-9133-b09ef48b4279"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:54.869219 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.869202 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-service-ca-bundle" pod "468f44ff-aa39-4909-9133-b09ef48b4279" (UID: "468f44ff-aa39-4909-9133-b09ef48b4279"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:54.870997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.870975 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468f44ff-aa39-4909-9133-b09ef48b4279-kube-api-access-2hfc7" pod "468f44ff-aa39-4909-9133-b09ef48b4279" (UID: "468f44ff-aa39-4909-9133-b09ef48b4279"). InnerVolumeSpecName "kube-api-access-2hfc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:54.969705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.969672 4862 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-trusted-ca-bundle\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:00:54.969705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.969700 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hfc7\" (UniqueName: \"kubernetes.io/projected/468f44ff-aa39-4909-9133-b09ef48b4279-kube-api-access-2hfc7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:00:54.969705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.969709 4862 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/468f44ff-aa39-4909-9133-b09ef48b4279-service-ca-bundle\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:00:54.969907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:54.969718 4862 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/468f44ff-aa39-4909-9133-b09ef48b4279-archives-path\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:00:55.737928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:55.737896 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" event={"ID":"468f44ff-aa39-4909-9133-b09ef48b4279","Type":"ContainerDied","Data":"b499a0d9b85e4bf52d826598aebc6d3f3e7c7629f29a208df81d8aff8cae753f"} Oct 30 18:00:55.737928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:55.737920 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fn4jd-xgpg5" Oct 30 18:00:55.738215 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:00:55.737940 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b499a0d9b85e4bf52d826598aebc6d3f3e7c7629f29a208df81d8aff8cae753f" Oct 30 18:01:19.869804 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:01:19.869768 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:01:49.867660 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:01:49.867620 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:02:19.868504 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:02:19.868465 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:02:29.939038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.939009 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9"] Oct 30 18:02:29.939541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.939324 4862 cpu_manager.go:414] "containerMap: removing container" podUID="468f44ff-aa39-4909-9133-b09ef48b4279" containerName="insights-gathering" Oct 30 18:02:29.939541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.939335 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468f44ff-aa39-4909-9133-b09ef48b4279" containerName="insights-gathering" Oct 30 18:02:29.939541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.939405 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="468f44ff-aa39-4909-9133-b09ef48b4279" containerName="insights-gathering" Oct 30 18:02:29.942229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.942213 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:29.955144 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.955128 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:02:29.955671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.955652 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:02:29.965800 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.965763 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:02:29.965800 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.965782 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:02:29.965925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.965804 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4jrp2\"" Oct 30 18:02:29.968016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:29.967990 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9"] Oct 30 18:02:30.017656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.017637 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.017757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.017673 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.017757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.017694 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.017757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.017715 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-kube-api-access-gc4ps\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.017859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.017829 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.017859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.017850 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.017922 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.017875 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d490abc0-ff82-4959-864e-54e0266b3ff8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.118176 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.118154 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.118285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.118180 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.118285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.118207 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d490abc0-ff82-4959-864e-54e0266b3ff8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.118285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.118233 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.118285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.118283 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.118492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.118420 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.118492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.118465 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-kube-api-access-gc4ps\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.119008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.118984 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.120613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.120594 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d490abc0-ff82-4959-864e-54e0266b3ff8-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.120807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.120786 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.121168 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.121146 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.121168 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.121158 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.135688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.135661 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.136096 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.136075 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-kube-api-access-gc4ps\") pod \"istiod-openshift-gateway-7876bfcb74-57kj9\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.253342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.253288 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:30.559475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:30.559420 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9"] Oct 30 18:02:30.564145 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:02:30.564120 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd490abc0_ff82_4959_864e_54e0266b3ff8.slice/crio-165a05f22f052e4836ba6e336872799eda5afa00d977f23805f41f4bd27cdfb8 WatchSource:0}: Error finding container 165a05f22f052e4836ba6e336872799eda5afa00d977f23805f41f4bd27cdfb8: Status 404 returned error can't find the container with id 165a05f22f052e4836ba6e336872799eda5afa00d977f23805f41f4bd27cdfb8 Oct 30 18:02:31.068713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:31.068687 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" event={"ID":"d490abc0-ff82-4959-864e-54e0266b3ff8","Type":"ContainerStarted","Data":"165a05f22f052e4836ba6e336872799eda5afa00d977f23805f41f4bd27cdfb8"} Oct 30 18:02:33.184179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:33.184149 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:02:33.184453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:33.184214 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:02:34.081553 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:34.081514 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" event={"ID":"d490abc0-ff82-4959-864e-54e0266b3ff8","Type":"ContainerStarted","Data":"0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba"} Oct 30 18:02:34.081720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:34.081669 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:34.083300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:34.083276 4862 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-57kj9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:02:34.083416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:34.083322 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" podUID="d490abc0-ff82-4959-864e-54e0266b3ff8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:02:35.085607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:35.085581 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:02:35.145483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:02:35.145438 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" podStartSLOduration=3.528295314 podStartE2EDuration="6.145418854s" totalImagesPullingTime="2.61712354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:34.127052723 +0000 UTC m=+253.176097638" watchObservedRunningTime="2026-10-30 18:02:35.145418854 +0000 UTC m=+254.194463766" Oct 30 18:02:49.868640 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:02:49.868605 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:03:19.869004 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:03:19.868963 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:03:21.075515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:21.075489 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:03:21.076090 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:21.076067 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:03:26.786614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.786579 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr"] Oct 30 18:03:26.796431 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.796401 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:26.812604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.812578 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr"] Oct 30 18:03:26.879954 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.879921 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:03:26.880115 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.880067 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:03:26.880336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.880315 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7dbhx\"" Oct 30 18:03:26.880436 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.880368 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:03:26.880436 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.880377 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:03:26.880511 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.880451 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:03:26.908044 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.908014 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-tmp\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:26.908140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.908048 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-metrics-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:26.908140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.908088 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:26.908233 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.908165 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-manifests\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:26.908296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.908228 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:26.908344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:26.908316 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4gss\" (UniqueName: \"kubernetes.io/projected/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-kube-api-access-d4gss\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.009397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.009373 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.009491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.009401 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-manifests\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.009491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.009426 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.009491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.009450 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gss\" (UniqueName: \"kubernetes.io/projected/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-kube-api-access-d4gss\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.009491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.009486 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-tmp\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.009681 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.009506 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-metrics-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.009846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.009828 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-manifests\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.012246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.012220 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-tmp\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.012378 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.012358 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-metrics-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.012508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.012489 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-webhook-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.012563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.012506 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.032688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.032661 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4gss\" (UniqueName: \"kubernetes.io/projected/6dcf5030-2a0c-4c27-bd6b-d8ee68d55009-kube-api-access-d4gss\") pod \"opendatahub-operator-controller-manager-58c455bdd6-7g2mr\" (UID: \"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.110453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.110415 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:27.466700 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.466626 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr"] Oct 30 18:03:27.470363 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:03:27.470327 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcf5030_2a0c_4c27_bd6b_d8ee68d55009.slice/crio-06df5ad90af3c8c8ef49bfec5ff1edf807b65e5afd25bc7e9eca637c9ccfe1fe WatchSource:0}: Error finding container 06df5ad90af3c8c8ef49bfec5ff1edf807b65e5afd25bc7e9eca637c9ccfe1fe: Status 404 returned error can't find the container with id 06df5ad90af3c8c8ef49bfec5ff1edf807b65e5afd25bc7e9eca637c9ccfe1fe Oct 30 18:03:27.471947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:27.471929 4862 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:03:28.275602 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:28.275566 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" event={"ID":"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009","Type":"ContainerStarted","Data":"06df5ad90af3c8c8ef49bfec5ff1edf807b65e5afd25bc7e9eca637c9ccfe1fe"} Oct 30 18:03:31.289330 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:31.289203 4862 generic.go:356] "Generic (PLEG): container finished" podID="6dcf5030-2a0c-4c27-bd6b-d8ee68d55009" containerID="708bd026439a6e389303dc6c18e6c468aef4899de8c389a29628dacd2d9c53be" exitCode=0 Oct 30 18:03:31.289330 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:31.289246 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" event={"ID":"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009","Type":"ContainerDied","Data":"708bd026439a6e389303dc6c18e6c468aef4899de8c389a29628dacd2d9c53be"} Oct 30 18:03:32.296532 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:32.296494 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" event={"ID":"6dcf5030-2a0c-4c27-bd6b-d8ee68d55009","Type":"ContainerStarted","Data":"84516f439ac7f3a44c8bc27772f7aa7c71c7163034000f008960eaf2128af2c9"} Oct 30 18:03:32.297011 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:32.296639 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:32.411833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:32.411776 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" podStartSLOduration=2.726739344 podStartE2EDuration="6.411761726s" totalImagesPullingTime="3.231274431s" totalInitContainerRuntime="453.747951ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:32.410048151 +0000 UTC m=+311.459093064" watchObservedRunningTime="2026-10-30 18:03:32.411761726 +0000 UTC m=+311.460806702" Oct 30 18:03:43.302994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:03:43.302966 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58c455bdd6-7g2mr" Oct 30 18:03:49.869063 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:03:49.869022 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:04:02.125524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.125487 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb"] Oct 30 18:04:02.129040 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.129020 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.163658 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.163622 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 18:04:02.163658 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.163622 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2c4ww\"" Oct 30 18:04:02.163884 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.163677 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 18:04:02.165601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.165573 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb"] Oct 30 18:04:02.275570 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.275533 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttpgg\" (UniqueName: \"kubernetes.io/projected/c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e-kube-api-access-ttpgg\") pod \"kube-auth-proxy-7ffc6d98d7-wrwtb\" (UID: \"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.275733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.275599 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e-tls-certs\") pod \"kube-auth-proxy-7ffc6d98d7-wrwtb\" (UID: \"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.275733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.275706 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e-tmp\") pod \"kube-auth-proxy-7ffc6d98d7-wrwtb\" (UID: \"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.377016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.376941 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e-tmp\") pod \"kube-auth-proxy-7ffc6d98d7-wrwtb\" (UID: \"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.377016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.377009 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttpgg\" (UniqueName: \"kubernetes.io/projected/c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e-kube-api-access-ttpgg\") pod \"kube-auth-proxy-7ffc6d98d7-wrwtb\" (UID: \"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.377203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.377071 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e-tls-certs\") pod \"kube-auth-proxy-7ffc6d98d7-wrwtb\" (UID: \"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.379357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.379315 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e-tmp\") pod \"kube-auth-proxy-7ffc6d98d7-wrwtb\" (UID: \"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.379668 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.379645 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e-tls-certs\") pod \"kube-auth-proxy-7ffc6d98d7-wrwtb\" (UID: \"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.407128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.407105 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttpgg\" (UniqueName: \"kubernetes.io/projected/c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e-kube-api-access-ttpgg\") pod \"kube-auth-proxy-7ffc6d98d7-wrwtb\" (UID: \"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e\") " pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.441432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.441402 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" Oct 30 18:04:02.737100 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:02.737030 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb"] Oct 30 18:04:02.740504 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:04:02.740475 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13dd46c_b1fa_4044_b9d0_6c2a1a5f1a6e.slice/crio-70572798260b40f8d787cc086777509d68044d577c7131f4f407aa8508878d50 WatchSource:0}: Error finding container 70572798260b40f8d787cc086777509d68044d577c7131f4f407aa8508878d50: Status 404 returned error can't find the container with id 70572798260b40f8d787cc086777509d68044d577c7131f4f407aa8508878d50 Oct 30 18:04:03.423806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:03.423772 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" event={"ID":"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e","Type":"ContainerStarted","Data":"70572798260b40f8d787cc086777509d68044d577c7131f4f407aa8508878d50"} Oct 30 18:04:05.434256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:05.434173 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" event={"ID":"c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e","Type":"ContainerStarted","Data":"d4a4e51eb4391e51910a2bbc74ded7daee180337216b07ccc1b126ac05014957"} Oct 30 18:04:05.471638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:05.471594 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ffc6d98d7-wrwtb" podStartSLOduration=1.089567095 podStartE2EDuration="3.471580961s" totalImagesPullingTime="2.382013866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:05.470327607 +0000 UTC m=+344.519372519" watchObservedRunningTime="2026-10-30 18:04:05.471580961 +0000 UTC m=+344.520625875" Oct 30 18:04:19.870059 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:04:19.870015 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:04:23.043530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.043500 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h44fj"] Oct 30 18:04:23.048139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.048124 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" Oct 30 18:04:23.052768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.052746 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:04:23.055839 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.055822 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:23.065231 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.065207 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h44fj"] Oct 30 18:04:23.074598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.074575 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:04:23.074702 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.074651 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:04:23.074702 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.074693 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t6wvx\"" Oct 30 18:04:23.089234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.089214 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:04:23.131217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.131196 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:04:23.164080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.164060 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 18:04:23.164354 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.164340 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ks7zb\"" Oct 30 18:04:23.234724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.234702 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3204144-0813-45d0-a202-acb52f0aeaa1-config-file\") pod \"limitador-limitador-7c9b87d574-rlk29\" (UID: \"f3204144-0813-45d0-a202-acb52f0aeaa1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:23.234804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.234754 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl89z\" (UniqueName: \"kubernetes.io/projected/e6f7d7ef-6012-4fdc-865d-6a642f2ac992-kube-api-access-tl89z\") pod \"authorino-56c6c9ddd5-h44fj\" (UID: \"e6f7d7ef-6012-4fdc-865d-6a642f2ac992\") " pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" Oct 30 18:04:23.234856 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.234840 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z769\" (UniqueName: \"kubernetes.io/projected/f3204144-0813-45d0-a202-acb52f0aeaa1-kube-api-access-2z769\") pod \"limitador-limitador-7c9b87d574-rlk29\" (UID: \"f3204144-0813-45d0-a202-acb52f0aeaa1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:23.335925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.335892 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3204144-0813-45d0-a202-acb52f0aeaa1-config-file\") pod \"limitador-limitador-7c9b87d574-rlk29\" (UID: \"f3204144-0813-45d0-a202-acb52f0aeaa1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:23.336017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.335950 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl89z\" (UniqueName: \"kubernetes.io/projected/e6f7d7ef-6012-4fdc-865d-6a642f2ac992-kube-api-access-tl89z\") pod \"authorino-56c6c9ddd5-h44fj\" (UID: \"e6f7d7ef-6012-4fdc-865d-6a642f2ac992\") " pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" Oct 30 18:04:23.336017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.335977 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z769\" (UniqueName: \"kubernetes.io/projected/f3204144-0813-45d0-a202-acb52f0aeaa1-kube-api-access-2z769\") pod \"limitador-limitador-7c9b87d574-rlk29\" (UID: \"f3204144-0813-45d0-a202-acb52f0aeaa1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:23.336492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.336475 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f3204144-0813-45d0-a202-acb52f0aeaa1-config-file\") pod \"limitador-limitador-7c9b87d574-rlk29\" (UID: \"f3204144-0813-45d0-a202-acb52f0aeaa1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:23.353856 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.353835 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z769\" (UniqueName: \"kubernetes.io/projected/f3204144-0813-45d0-a202-acb52f0aeaa1-kube-api-access-2z769\") pod \"limitador-limitador-7c9b87d574-rlk29\" (UID: \"f3204144-0813-45d0-a202-acb52f0aeaa1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:23.353944 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.353918 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl89z\" (UniqueName: \"kubernetes.io/projected/e6f7d7ef-6012-4fdc-865d-6a642f2ac992-kube-api-access-tl89z\") pod \"authorino-56c6c9ddd5-h44fj\" (UID: \"e6f7d7ef-6012-4fdc-865d-6a642f2ac992\") " pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" Oct 30 18:04:23.370681 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.370662 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" Oct 30 18:04:23.381575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.381556 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:23.680757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.680690 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h44fj"] Oct 30 18:04:23.685175 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:04:23.685141 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6f7d7ef_6012_4fdc_865d_6a642f2ac992.slice/crio-9baf805b33e4ac16729ef6673128713534e767f43245c244da53e5968c1be782 WatchSource:0}: Error finding container 9baf805b33e4ac16729ef6673128713534e767f43245c244da53e5968c1be782: Status 404 returned error can't find the container with id 9baf805b33e4ac16729ef6673128713534e767f43245c244da53e5968c1be782 Oct 30 18:04:23.694127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:23.694103 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:04:23.698522 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:04:23.698497 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3204144_0813_45d0_a202_acb52f0aeaa1.slice/crio-32add6f7e10e4731c62533ac384ab256decb0d6536087da9f2cebe0b1013379a WatchSource:0}: Error finding container 32add6f7e10e4731c62533ac384ab256decb0d6536087da9f2cebe0b1013379a: Status 404 returned error can't find the container with id 32add6f7e10e4731c62533ac384ab256decb0d6536087da9f2cebe0b1013379a Oct 30 18:04:24.514225 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:24.514150 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" event={"ID":"f3204144-0813-45d0-a202-acb52f0aeaa1","Type":"ContainerStarted","Data":"32add6f7e10e4731c62533ac384ab256decb0d6536087da9f2cebe0b1013379a"} Oct 30 18:04:24.515989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:24.515958 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" event={"ID":"e6f7d7ef-6012-4fdc-865d-6a642f2ac992","Type":"ContainerStarted","Data":"9baf805b33e4ac16729ef6673128713534e767f43245c244da53e5968c1be782"} Oct 30 18:04:26.528818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:26.528253 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" event={"ID":"f3204144-0813-45d0-a202-acb52f0aeaa1","Type":"ContainerStarted","Data":"d5a29eb64665545866bca8813487f9e42c55ab083afccb66ecbbc7e325121a66"} Oct 30 18:04:26.528818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:26.528451 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:26.529765 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:26.529742 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" event={"ID":"e6f7d7ef-6012-4fdc-865d-6a642f2ac992","Type":"ContainerStarted","Data":"188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796"} Oct 30 18:04:26.569242 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:26.569196 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" podStartSLOduration=1.028244312 podStartE2EDuration="3.5691862s" totalImagesPullingTime="2.540941888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:26.567722835 +0000 UTC m=+365.616767749" watchObservedRunningTime="2026-10-30 18:04:26.5691862 +0000 UTC m=+365.618231112" Oct 30 18:04:26.611439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:26.611393 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" podStartSLOduration=1.058092232 podStartE2EDuration="3.61138019s" totalImagesPullingTime="2.553287958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:26.611095096 +0000 UTC m=+365.660140010" watchObservedRunningTime="2026-10-30 18:04:26.61138019 +0000 UTC m=+365.660425102" Oct 30 18:04:37.534373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:04:37.534347 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-rlk29" Oct 30 18:04:49.867903 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:04:49.867872 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:05:02.650450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.650368 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95"] Oct 30 18:05:02.652736 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.652715 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:02.684689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.684661 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 18:05:02.684879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.684860 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lcds2\"" Oct 30 18:05:02.685990 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.685964 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95"] Oct 30 18:05:02.730967 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.730934 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptht\" (UniqueName: \"kubernetes.io/projected/49c5b13b-46cc-4116-a59d-ca7abeecb539-kube-api-access-rptht\") pod \"llmisvc-controller-manager-869d5dcc5d-nwq95\" (UID: \"49c5b13b-46cc-4116-a59d-ca7abeecb539\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:02.730967 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.730970 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49c5b13b-46cc-4116-a59d-ca7abeecb539-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nwq95\" (UID: \"49c5b13b-46cc-4116-a59d-ca7abeecb539\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:02.734756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.734736 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:05:02.734881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.734868 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:05:02.747084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.747059 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-9jr6v"] Oct 30 18:05:02.749305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.749290 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:02.777143 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.777039 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9jr6v"] Oct 30 18:05:02.831579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.831543 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d0e911-850b-46bc-aabe-237660e9dbaa-metrics-certs\") pod \"odh-model-controller-769875db56-9jr6v\" (UID: \"e7d0e911-850b-46bc-aabe-237660e9dbaa\") " pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:02.831751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.831589 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59x5s\" (UniqueName: \"kubernetes.io/projected/e7d0e911-850b-46bc-aabe-237660e9dbaa-kube-api-access-59x5s\") pod \"odh-model-controller-769875db56-9jr6v\" (UID: \"e7d0e911-850b-46bc-aabe-237660e9dbaa\") " pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:02.831751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.831664 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptht\" (UniqueName: \"kubernetes.io/projected/49c5b13b-46cc-4116-a59d-ca7abeecb539-kube-api-access-rptht\") pod \"llmisvc-controller-manager-869d5dcc5d-nwq95\" (UID: \"49c5b13b-46cc-4116-a59d-ca7abeecb539\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:02.831751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.831699 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49c5b13b-46cc-4116-a59d-ca7abeecb539-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nwq95\" (UID: \"49c5b13b-46cc-4116-a59d-ca7abeecb539\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:02.831751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.831717 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7d0e911-850b-46bc-aabe-237660e9dbaa-cert\") pod \"odh-model-controller-769875db56-9jr6v\" (UID: \"e7d0e911-850b-46bc-aabe-237660e9dbaa\") " pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:02.834245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.834221 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49c5b13b-46cc-4116-a59d-ca7abeecb539-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nwq95\" (UID: \"49c5b13b-46cc-4116-a59d-ca7abeecb539\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:02.836640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.836621 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 18:05:02.836640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.836632 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 18:05:02.836640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.836624 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5cz6f\"" Oct 30 18:05:02.857680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.857651 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptht\" (UniqueName: \"kubernetes.io/projected/49c5b13b-46cc-4116-a59d-ca7abeecb539-kube-api-access-rptht\") pod \"llmisvc-controller-manager-869d5dcc5d-nwq95\" (UID: \"49c5b13b-46cc-4116-a59d-ca7abeecb539\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:02.932310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.932245 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d0e911-850b-46bc-aabe-237660e9dbaa-metrics-certs\") pod \"odh-model-controller-769875db56-9jr6v\" (UID: \"e7d0e911-850b-46bc-aabe-237660e9dbaa\") " pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:02.932428 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.932329 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59x5s\" (UniqueName: \"kubernetes.io/projected/e7d0e911-850b-46bc-aabe-237660e9dbaa-kube-api-access-59x5s\") pod \"odh-model-controller-769875db56-9jr6v\" (UID: \"e7d0e911-850b-46bc-aabe-237660e9dbaa\") " pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:02.932428 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.932381 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7d0e911-850b-46bc-aabe-237660e9dbaa-cert\") pod \"odh-model-controller-769875db56-9jr6v\" (UID: \"e7d0e911-850b-46bc-aabe-237660e9dbaa\") " pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:02.935464 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.935439 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7d0e911-850b-46bc-aabe-237660e9dbaa-cert\") pod \"odh-model-controller-769875db56-9jr6v\" (UID: \"e7d0e911-850b-46bc-aabe-237660e9dbaa\") " pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:02.935584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.935439 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7d0e911-850b-46bc-aabe-237660e9dbaa-metrics-certs\") pod \"odh-model-controller-769875db56-9jr6v\" (UID: \"e7d0e911-850b-46bc-aabe-237660e9dbaa\") " pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:02.964202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.964181 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:02.968763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:02.968746 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59x5s\" (UniqueName: \"kubernetes.io/projected/e7d0e911-850b-46bc-aabe-237660e9dbaa-kube-api-access-59x5s\") pod \"odh-model-controller-769875db56-9jr6v\" (UID: \"e7d0e911-850b-46bc-aabe-237660e9dbaa\") " pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:03.061223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:03.061200 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:03.257359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:03.257291 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95"] Oct 30 18:05:03.260975 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:05:03.260951 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49c5b13b_46cc_4116_a59d_ca7abeecb539.slice/crio-309393e2e0f62e71f2efe8b12b15fac067d3d8c766b4ae0a9ad0e2254320f951 WatchSource:0}: Error finding container 309393e2e0f62e71f2efe8b12b15fac067d3d8c766b4ae0a9ad0e2254320f951: Status 404 returned error can't find the container with id 309393e2e0f62e71f2efe8b12b15fac067d3d8c766b4ae0a9ad0e2254320f951 Oct 30 18:05:03.351763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:03.351735 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9jr6v"] Oct 30 18:05:03.355834 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:05:03.355810 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d0e911_850b_46bc_aabe_237660e9dbaa.slice/crio-23eb8b134deaf56dc6d6edb60a53a8efbccde58632896219e26fc434899b8bf4 WatchSource:0}: Error finding container 23eb8b134deaf56dc6d6edb60a53a8efbccde58632896219e26fc434899b8bf4: Status 404 returned error can't find the container with id 23eb8b134deaf56dc6d6edb60a53a8efbccde58632896219e26fc434899b8bf4 Oct 30 18:05:03.677484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:03.677453 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9jr6v" event={"ID":"e7d0e911-850b-46bc-aabe-237660e9dbaa","Type":"ContainerStarted","Data":"23eb8b134deaf56dc6d6edb60a53a8efbccde58632896219e26fc434899b8bf4"} Oct 30 18:05:03.678629 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:03.678605 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" event={"ID":"49c5b13b-46cc-4116-a59d-ca7abeecb539","Type":"ContainerStarted","Data":"309393e2e0f62e71f2efe8b12b15fac067d3d8c766b4ae0a9ad0e2254320f951"} Oct 30 18:05:06.696009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:06.695976 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9jr6v" event={"ID":"e7d0e911-850b-46bc-aabe-237660e9dbaa","Type":"ContainerStarted","Data":"abcd818776fd629f17400bc7d0020535e1593189da8bbfbf408eb0bde5a78790"} Oct 30 18:05:06.696475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:06.696150 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:06.697471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:06.697452 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" event={"ID":"49c5b13b-46cc-4116-a59d-ca7abeecb539","Type":"ContainerStarted","Data":"37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a"} Oct 30 18:05:06.697582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:06.697568 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:06.734451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:06.734410 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-9jr6v" podStartSLOduration=2.08761344 podStartE2EDuration="4.73439797s" totalImagesPullingTime="2.64678453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:06.733155137 +0000 UTC m=+405.782200051" watchObservedRunningTime="2026-10-30 18:05:06.73439797 +0000 UTC m=+405.783442883" Oct 30 18:05:06.771956 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:06.771914 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" podStartSLOduration=2.2633478399999998 podStartE2EDuration="4.771901521s" totalImagesPullingTime="2.508553681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:06.770562296 +0000 UTC m=+405.819607209" watchObservedRunningTime="2026-10-30 18:05:06.771901521 +0000 UTC m=+405.820946497" Oct 30 18:05:07.305847 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.305815 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9"] Oct 30 18:05:07.306083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.306054 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" podUID="d490abc0-ff82-4959-864e-54e0266b3ff8" containerName="discovery" containerID="cri-o://0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba" gracePeriod=30 Oct 30 18:05:07.549893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.549872 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:05:07.668951 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.668884 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-token\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-token\") pod \"d490abc0-ff82-4959-864e-54e0266b3ff8\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " Oct 30 18:05:07.668951 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.668949 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-ca-configmap\") pod \"d490abc0-ff82-4959-864e-54e0266b3ff8\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " Oct 30 18:05:07.669123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.668967 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-kubeconfig\") pod \"d490abc0-ff82-4959-864e-54e0266b3ff8\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " Oct 30 18:05:07.669123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.669100 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-kube-api-access-gc4ps\") pod \"d490abc0-ff82-4959-864e-54e0266b3ff8\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " Oct 30 18:05:07.669192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.669152 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-cacerts\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-cacerts\") pod \"d490abc0-ff82-4959-864e-54e0266b3ff8\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " Oct 30 18:05:07.669225 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.669194 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d490abc0-ff82-4959-864e-54e0266b3ff8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d490abc0-ff82-4959-864e-54e0266b3ff8-local-certs\") pod \"d490abc0-ff82-4959-864e-54e0266b3ff8\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " Oct 30 18:05:07.669304 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.669243 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-dns-cert\") pod \"d490abc0-ff82-4959-864e-54e0266b3ff8\" (UID: \"d490abc0-ff82-4959-864e-54e0266b3ff8\") " Oct 30 18:05:07.669472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.669448 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-ca-configmap" pod "d490abc0-ff82-4959-864e-54e0266b3ff8" (UID: "d490abc0-ff82-4959-864e-54e0266b3ff8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:07.669598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.669573 4862 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-ca-configmap\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.671624 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.671595 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-token" pod "d490abc0-ff82-4959-864e-54e0266b3ff8" (UID: "d490abc0-ff82-4959-864e-54e0266b3ff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:07.671624 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.671588 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d490abc0-ff82-4959-864e-54e0266b3ff8-local-certs" pod "d490abc0-ff82-4959-864e-54e0266b3ff8" (UID: "d490abc0-ff82-4959-864e-54e0266b3ff8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:07.671824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.671800 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-kube-api-access-gc4ps" pod "d490abc0-ff82-4959-864e-54e0266b3ff8" (UID: "d490abc0-ff82-4959-864e-54e0266b3ff8"). InnerVolumeSpecName "kube-api-access-gc4ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:07.671944 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.671923 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-kubeconfig" pod "d490abc0-ff82-4959-864e-54e0266b3ff8" (UID: "d490abc0-ff82-4959-864e-54e0266b3ff8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:07.672048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.672027 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-cacerts" pod "d490abc0-ff82-4959-864e-54e0266b3ff8" (UID: "d490abc0-ff82-4959-864e-54e0266b3ff8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:07.672097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.672083 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-dns-cert" pod "d490abc0-ff82-4959-864e-54e0266b3ff8" (UID: "d490abc0-ff82-4959-864e-54e0266b3ff8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:07.703236 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.703212 4862 generic.go:356] "Generic (PLEG): container finished" podID="d490abc0-ff82-4959-864e-54e0266b3ff8" containerID="0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba" exitCode=0 Oct 30 18:05:07.703603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.703284 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" Oct 30 18:05:07.703603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.703292 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" event={"ID":"d490abc0-ff82-4959-864e-54e0266b3ff8","Type":"ContainerDied","Data":"0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba"} Oct 30 18:05:07.703603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.703378 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9" event={"ID":"d490abc0-ff82-4959-864e-54e0266b3ff8","Type":"ContainerDied","Data":"165a05f22f052e4836ba6e336872799eda5afa00d977f23805f41f4bd27cdfb8"} Oct 30 18:05:07.703603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.703405 4862 scope.go:122] "RemoveContainer" containerID="0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba" Oct 30 18:05:07.711660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.711646 4862 scope.go:122] "RemoveContainer" containerID="0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba" Oct 30 18:05:07.711921 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:05:07.711903 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba\": container with ID starting with 0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba not found: ID does not exist" containerID="0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba" Oct 30 18:05:07.711963 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.711928 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba"} err="failed to get container status \"0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba\": rpc error: code = NotFound desc = could not find container \"0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba\": container with ID starting with 0bcaf696c8777cb5edca6bcfb72b982be67ed3bae90ae637585c9bcf8f3380ba not found: ID does not exist" Oct 30 18:05:07.750358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.750334 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9"] Oct 30 18:05:07.756908 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.756889 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-57kj9"] Oct 30 18:05:07.770167 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.770146 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.770167 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.770165 4862 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-kubeconfig\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.770370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.770173 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc4ps\" (UniqueName: \"kubernetes.io/projected/d490abc0-ff82-4959-864e-54e0266b3ff8-kube-api-access-gc4ps\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.770370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.770183 4862 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-cacerts\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.770370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.770193 4862 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d490abc0-ff82-4959-864e-54e0266b3ff8-local-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.770370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:07.770204 4862 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d490abc0-ff82-4959-864e-54e0266b3ff8-istio-csr-dns-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:08.693933 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.693898 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f"] Oct 30 18:05:08.694246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.694217 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d490abc0-ff82-4959-864e-54e0266b3ff8" containerName="discovery" Oct 30 18:05:08.694246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.694236 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d490abc0-ff82-4959-864e-54e0266b3ff8" containerName="discovery" Oct 30 18:05:08.694473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.694380 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d490abc0-ff82-4959-864e-54e0266b3ff8" containerName="discovery" Oct 30 18:05:08.698577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.698556 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.715475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.715459 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-mdtjb\"" Oct 30 18:05:08.715758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.715484 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 18:05:08.730422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.730397 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f"] Oct 30 18:05:08.778499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.778472 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7afa1539-e989-4bdd-bdda-eccf8ae776c3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.778598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.778509 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7afa1539-e989-4bdd-bdda-eccf8ae776c3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.778598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.778538 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wf6\" (UniqueName: \"kubernetes.io/projected/7afa1539-e989-4bdd-bdda-eccf8ae776c3-kube-api-access-v6wf6\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.778674 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.778658 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7afa1539-e989-4bdd-bdda-eccf8ae776c3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.793860 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.793839 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 18:05:08.794017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.793999 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 18:05:08.879824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.879800 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7afa1539-e989-4bdd-bdda-eccf8ae776c3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.879920 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.879832 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wf6\" (UniqueName: \"kubernetes.io/projected/7afa1539-e989-4bdd-bdda-eccf8ae776c3-kube-api-access-v6wf6\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.879920 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.879855 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7afa1539-e989-4bdd-bdda-eccf8ae776c3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.879986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.879929 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7afa1539-e989-4bdd-bdda-eccf8ae776c3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.880561 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.880542 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7afa1539-e989-4bdd-bdda-eccf8ae776c3-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.880803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.880659 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7afa1539-e989-4bdd-bdda-eccf8ae776c3-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.882453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.882433 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7afa1539-e989-4bdd-bdda-eccf8ae776c3-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:08.903328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:08.903306 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wf6\" (UniqueName: \"kubernetes.io/projected/7afa1539-e989-4bdd-bdda-eccf8ae776c3-kube-api-access-v6wf6\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7rt9f\" (UID: \"7afa1539-e989-4bdd-bdda-eccf8ae776c3\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:09.011985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:09.011931 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:09.062152 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:09.062116 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d490abc0-ff82-4959-864e-54e0266b3ff8" path="/var/lib/kubelet/pods/d490abc0-ff82-4959-864e-54e0266b3ff8/volumes" Oct 30 18:05:09.324480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:09.324450 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f"] Oct 30 18:05:09.329712 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:05:09.329672 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afa1539_e989_4bdd_bdda_eccf8ae776c3.slice/crio-09938f22ef9bd22ed31bf58dc33298e53585b479ec1332f11edb398a1a745dfe WatchSource:0}: Error finding container 09938f22ef9bd22ed31bf58dc33298e53585b479ec1332f11edb398a1a745dfe: Status 404 returned error can't find the container with id 09938f22ef9bd22ed31bf58dc33298e53585b479ec1332f11edb398a1a745dfe Oct 30 18:05:09.714680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:09.714605 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" event={"ID":"7afa1539-e989-4bdd-bdda-eccf8ae776c3","Type":"ContainerStarted","Data":"09938f22ef9bd22ed31bf58dc33298e53585b479ec1332f11edb398a1a745dfe"} Oct 30 18:05:17.706047 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:17.706015 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-9jr6v" Oct 30 18:05:19.876290 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:05:19.876240 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:05:20.774893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:20.774858 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" event={"ID":"7afa1539-e989-4bdd-bdda-eccf8ae776c3","Type":"ContainerStarted","Data":"317713d55750e7ba0285d15424bdce2ad03d630c090d4117cd9510d4e074dc64"} Oct 30 18:05:20.775074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:20.774935 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:20.836412 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:20.836365 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" podStartSLOduration=2.215499868 podStartE2EDuration="12.836352174s" totalImagesPullingTime="10.620852306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:20.834560762 +0000 UTC m=+419.883605676" watchObservedRunningTime="2026-10-30 18:05:20.836352174 +0000 UTC m=+419.885397087" Oct 30 18:05:31.785437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:31.785410 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7rt9f" Oct 30 18:05:36.544171 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.544135 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5"] Oct 30 18:05:36.547472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.547456 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.574357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.574332 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:05:36.574485 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.574366 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 18:05:36.574485 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.574410 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:05:36.574485 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.574410 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:05:36.579079 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.579061 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5"] Oct 30 18:05:36.725241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.725217 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6mh\" (UniqueName: \"kubernetes.io/projected/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-kube-api-access-9f6mh\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.725375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.725277 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-metrics-certs\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.725375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.725344 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-maas-api-tls\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.725375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.725361 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-openshift-service-ca\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.826482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.826460 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-metrics-certs\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.826586 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.826498 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-maas-api-tls\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.826586 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.826515 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-openshift-service-ca\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.826586 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.826547 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6mh\" (UniqueName: \"kubernetes.io/projected/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-kube-api-access-9f6mh\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.827227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.827205 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-openshift-service-ca\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.828979 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.828962 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-metrics-certs\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.829039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.829010 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-maas-api-tls\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.848089 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.848067 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6mh\" (UniqueName: \"kubernetes.io/projected/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-kube-api-access-9f6mh\") pod \"maas-api-65d69b8749-q2wd5\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:36.859925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:36.859905 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:37.167209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:37.167121 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5"] Oct 30 18:05:37.170955 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:05:37.170929 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode624d37e_68a2_4fb1_ad1f_5dc5c77db2f0.slice/crio-684fad1cd7de9bf450f02d5e8823cb0a9e74aade8fe8c603f0a4979a77dda711 WatchSource:0}: Error finding container 684fad1cd7de9bf450f02d5e8823cb0a9e74aade8fe8c603f0a4979a77dda711: Status 404 returned error can't find the container with id 684fad1cd7de9bf450f02d5e8823cb0a9e74aade8fe8c603f0a4979a77dda711 Oct 30 18:05:37.706065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:37.706044 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:05:37.846610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:37.846574 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" event={"ID":"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0","Type":"ContainerStarted","Data":"684fad1cd7de9bf450f02d5e8823cb0a9e74aade8fe8c603f0a4979a77dda711"} Oct 30 18:05:38.678582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:38.677934 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:05:40.861497 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:40.861458 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" event={"ID":"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0","Type":"ContainerStarted","Data":"8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0"} Oct 30 18:05:40.861967 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:40.861603 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:40.901166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:40.901124 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" podStartSLOduration=2.069595795 podStartE2EDuration="4.901111446s" totalImagesPullingTime="2.831515651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:40.900667143 +0000 UTC m=+439.949712058" watchObservedRunningTime="2026-10-30 18:05:40.901111446 +0000 UTC m=+439.950156359" Oct 30 18:05:46.870831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:46.870803 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:47.623156 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.623127 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fz4zb"] Oct 30 18:05:47.722475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.722445 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fz4zb"] Oct 30 18:05:47.722617 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.722531 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fz4zb" Oct 30 18:05:47.816603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.816578 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55"] Oct 30 18:05:47.822637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.822610 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khvcv\" (UniqueName: \"kubernetes.io/projected/893cb954-0550-4f5f-a056-1b6254099352-kube-api-access-khvcv\") pod \"authorino-6f6944bcdd-fz4zb\" (UID: \"893cb954-0550-4f5f-a056-1b6254099352\") " pod="kuadrant-system/authorino-6f6944bcdd-fz4zb" Oct 30 18:05:47.842735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.842713 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fz4zb"] Oct 30 18:05:47.842831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.842812 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:47.842887 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:05:47.842868 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-khvcv], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-fz4zb" podUID="893cb954-0550-4f5f-a056-1b6254099352" Oct 30 18:05:47.848255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.848231 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55"] Oct 30 18:05:47.865757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.865740 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wjb5t\"" Oct 30 18:05:47.881420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.881367 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-d4mxd"] Oct 30 18:05:47.889673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.889653 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fz4zb" Oct 30 18:05:47.907087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.907062 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-d4mxd"] Oct 30 18:05:47.907185 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.907158 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:47.915480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.915464 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fz4zb" Oct 30 18:05:47.922982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.922962 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khvcv\" (UniqueName: \"kubernetes.io/projected/893cb954-0550-4f5f-a056-1b6254099352-kube-api-access-khvcv\") pod \"authorino-6f6944bcdd-fz4zb\" (UID: \"893cb954-0550-4f5f-a056-1b6254099352\") " pod="kuadrant-system/authorino-6f6944bcdd-fz4zb" Oct 30 18:05:47.942107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.942090 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khvcv\" (UniqueName: \"kubernetes.io/projected/893cb954-0550-4f5f-a056-1b6254099352-kube-api-access-khvcv\") pod \"authorino-6f6944bcdd-fz4zb\" (UID: \"893cb954-0550-4f5f-a056-1b6254099352\") " pod="kuadrant-system/authorino-6f6944bcdd-fz4zb" Oct 30 18:05:47.965842 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.965824 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 18:05:47.999593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:47.999569 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-d4mxd"] Oct 30 18:05:47.999773 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:05:47.999755 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vdrg7 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" podUID="74e87056-1721-40dc-95ba-f2166492c523" Oct 30 18:05:48.023890 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.023865 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ltrv\" (UniqueName: \"kubernetes.io/projected/7254fe79-25df-49c3-843c-51f10d54f5f4-kube-api-access-8ltrv\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.023981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.023929 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/74e87056-1721-40dc-95ba-f2166492c523-tls-cert\") pod \"authorino-6889cbd8c4-d4mxd\" (UID: \"74e87056-1721-40dc-95ba-f2166492c523\") " pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:48.024024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.023992 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7254fe79-25df-49c3-843c-51f10d54f5f4-openshift-service-ca\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.024024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.024017 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7254fe79-25df-49c3-843c-51f10d54f5f4-maas-api-tls\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.024094 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.024037 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254fe79-25df-49c3-843c-51f10d54f5f4-metrics-certs\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.024094 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.024060 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrg7\" (UniqueName: \"kubernetes.io/projected/74e87056-1721-40dc-95ba-f2166492c523-kube-api-access-vdrg7\") pod \"authorino-6889cbd8c4-d4mxd\" (UID: \"74e87056-1721-40dc-95ba-f2166492c523\") " pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:48.032980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.032956 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85779f77c6-shks7"] Oct 30 18:05:48.038593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.038578 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:05:48.059335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.059310 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85779f77c6-shks7"] Oct 30 18:05:48.125250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.125227 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893cb954-0550-4f5f-a056-1b6254099352-kube-api-access-khvcv\" (UniqueName: \"kubernetes.io/projected/893cb954-0550-4f5f-a056-1b6254099352-kube-api-access-khvcv\") pod \"893cb954-0550-4f5f-a056-1b6254099352\" (UID: \"893cb954-0550-4f5f-a056-1b6254099352\") " Oct 30 18:05:48.125411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.125387 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ltrv\" (UniqueName: \"kubernetes.io/projected/7254fe79-25df-49c3-843c-51f10d54f5f4-kube-api-access-8ltrv\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.125481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.125447 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/74e87056-1721-40dc-95ba-f2166492c523-tls-cert\") pod \"authorino-6889cbd8c4-d4mxd\" (UID: \"74e87056-1721-40dc-95ba-f2166492c523\") " pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:48.125481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.125466 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7254fe79-25df-49c3-843c-51f10d54f5f4-openshift-service-ca\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.125589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.125481 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7254fe79-25df-49c3-843c-51f10d54f5f4-maas-api-tls\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.125589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.125502 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254fe79-25df-49c3-843c-51f10d54f5f4-metrics-certs\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.125589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.125522 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrg7\" (UniqueName: \"kubernetes.io/projected/74e87056-1721-40dc-95ba-f2166492c523-kube-api-access-vdrg7\") pod \"authorino-6889cbd8c4-d4mxd\" (UID: \"74e87056-1721-40dc-95ba-f2166492c523\") " pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:48.126490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.126422 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7254fe79-25df-49c3-843c-51f10d54f5f4-openshift-service-ca\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.127843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.127812 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893cb954-0550-4f5f-a056-1b6254099352-kube-api-access-khvcv" pod "893cb954-0550-4f5f-a056-1b6254099352" (UID: "893cb954-0550-4f5f-a056-1b6254099352"). InnerVolumeSpecName "kube-api-access-khvcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:48.128169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.128141 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7254fe79-25df-49c3-843c-51f10d54f5f4-metrics-certs\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.128255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.128207 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/74e87056-1721-40dc-95ba-f2166492c523-tls-cert\") pod \"authorino-6889cbd8c4-d4mxd\" (UID: \"74e87056-1721-40dc-95ba-f2166492c523\") " pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:48.128372 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.128357 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7254fe79-25df-49c3-843c-51f10d54f5f4-maas-api-tls\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.143671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.143648 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ltrv\" (UniqueName: \"kubernetes.io/projected/7254fe79-25df-49c3-843c-51f10d54f5f4-kube-api-access-8ltrv\") pod \"maas-api-7b4d8998bc-4lr55\" (UID: \"7254fe79-25df-49c3-843c-51f10d54f5f4\") " pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.143961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.143942 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrg7\" (UniqueName: \"kubernetes.io/projected/74e87056-1721-40dc-95ba-f2166492c523-kube-api-access-vdrg7\") pod \"authorino-6889cbd8c4-d4mxd\" (UID: \"74e87056-1721-40dc-95ba-f2166492c523\") " pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:48.182746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.182723 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:48.226286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.226240 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whv6\" (UniqueName: \"kubernetes.io/projected/5c67be0f-89db-4e9c-a225-21f6bd526718-kube-api-access-4whv6\") pod \"authorino-85779f77c6-shks7\" (UID: \"5c67be0f-89db-4e9c-a225-21f6bd526718\") " pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:05:48.226392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.226376 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c67be0f-89db-4e9c-a225-21f6bd526718-tls-cert\") pod \"authorino-85779f77c6-shks7\" (UID: \"5c67be0f-89db-4e9c-a225-21f6bd526718\") " pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:05:48.226477 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.226468 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khvcv\" (UniqueName: \"kubernetes.io/projected/893cb954-0550-4f5f-a056-1b6254099352-kube-api-access-khvcv\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:48.327788 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.327760 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whv6\" (UniqueName: \"kubernetes.io/projected/5c67be0f-89db-4e9c-a225-21f6bd526718-kube-api-access-4whv6\") pod \"authorino-85779f77c6-shks7\" (UID: \"5c67be0f-89db-4e9c-a225-21f6bd526718\") " pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:05:48.327906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.327795 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c67be0f-89db-4e9c-a225-21f6bd526718-tls-cert\") pod \"authorino-85779f77c6-shks7\" (UID: \"5c67be0f-89db-4e9c-a225-21f6bd526718\") " pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:05:48.330313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.330295 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c67be0f-89db-4e9c-a225-21f6bd526718-tls-cert\") pod \"authorino-85779f77c6-shks7\" (UID: \"5c67be0f-89db-4e9c-a225-21f6bd526718\") " pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:05:48.345842 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.345818 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whv6\" (UniqueName: \"kubernetes.io/projected/5c67be0f-89db-4e9c-a225-21f6bd526718-kube-api-access-4whv6\") pod \"authorino-85779f77c6-shks7\" (UID: \"5c67be0f-89db-4e9c-a225-21f6bd526718\") " pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:05:48.352649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.352634 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:05:48.535564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.535513 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55"] Oct 30 18:05:48.539758 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:05:48.539731 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7254fe79_25df_49c3_843c_51f10d54f5f4.slice/crio-bfe1f32c8a8ea1cfea481423a03d7d37acdbb4dbd2221e5551490c9c667a0cda WatchSource:0}: Error finding container bfe1f32c8a8ea1cfea481423a03d7d37acdbb4dbd2221e5551490c9c667a0cda: Status 404 returned error can't find the container with id bfe1f32c8a8ea1cfea481423a03d7d37acdbb4dbd2221e5551490c9c667a0cda Oct 30 18:05:48.674309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.674216 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85779f77c6-shks7"] Oct 30 18:05:48.678824 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:05:48.678805 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c67be0f_89db_4e9c_a225_21f6bd526718.slice/crio-9416adaf0e9293b363cf539e6356080e6605aaa36f086fe0940565426c11b7dd WatchSource:0}: Error finding container 9416adaf0e9293b363cf539e6356080e6605aaa36f086fe0940565426c11b7dd: Status 404 returned error can't find the container with id 9416adaf0e9293b363cf539e6356080e6605aaa36f086fe0940565426c11b7dd Oct 30 18:05:48.894755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.894728 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85779f77c6-shks7" event={"ID":"5c67be0f-89db-4e9c-a225-21f6bd526718","Type":"ContainerStarted","Data":"9416adaf0e9293b363cf539e6356080e6605aaa36f086fe0940565426c11b7dd"} Oct 30 18:05:48.895945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.895920 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:48.896057 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.895949 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fz4zb" Oct 30 18:05:48.896057 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.895919 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" event={"ID":"7254fe79-25df-49c3-843c-51f10d54f5f4","Type":"ContainerStarted","Data":"bfe1f32c8a8ea1cfea481423a03d7d37acdbb4dbd2221e5551490c9c667a0cda"} Oct 30 18:05:48.923170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.923152 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:48.988630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.988574 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fz4zb"] Oct 30 18:05:48.991104 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:48.991082 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fz4zb"] Oct 30 18:05:49.032954 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.032928 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e87056-1721-40dc-95ba-f2166492c523-kube-api-access-vdrg7\" (UniqueName: \"kubernetes.io/projected/74e87056-1721-40dc-95ba-f2166492c523-kube-api-access-vdrg7\") pod \"74e87056-1721-40dc-95ba-f2166492c523\" (UID: \"74e87056-1721-40dc-95ba-f2166492c523\") " Oct 30 18:05:49.033058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.032966 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e87056-1721-40dc-95ba-f2166492c523-tls-cert\" (UniqueName: \"kubernetes.io/secret/74e87056-1721-40dc-95ba-f2166492c523-tls-cert\") pod \"74e87056-1721-40dc-95ba-f2166492c523\" (UID: \"74e87056-1721-40dc-95ba-f2166492c523\") " Oct 30 18:05:49.035145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.035123 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e87056-1721-40dc-95ba-f2166492c523-kube-api-access-vdrg7" pod "74e87056-1721-40dc-95ba-f2166492c523" (UID: "74e87056-1721-40dc-95ba-f2166492c523"). InnerVolumeSpecName "kube-api-access-vdrg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:49.035237 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.035180 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e87056-1721-40dc-95ba-f2166492c523-tls-cert" pod "74e87056-1721-40dc-95ba-f2166492c523" (UID: "74e87056-1721-40dc-95ba-f2166492c523"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:49.060001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.059975 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893cb954-0550-4f5f-a056-1b6254099352" path="/var/lib/kubelet/pods/893cb954-0550-4f5f-a056-1b6254099352/volumes" Oct 30 18:05:49.133411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.133391 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdrg7\" (UniqueName: \"kubernetes.io/projected/74e87056-1721-40dc-95ba-f2166492c523-kube-api-access-vdrg7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.133504 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.133414 4862 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/74e87056-1721-40dc-95ba-f2166492c523-tls-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:49.873352 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:05:49.873317 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:05:49.906351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.906246 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85779f77c6-shks7" event={"ID":"5c67be0f-89db-4e9c-a225-21f6bd526718","Type":"ContainerStarted","Data":"0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954"} Oct 30 18:05:49.908007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.907982 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-d4mxd" Oct 30 18:05:49.908007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.907994 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" event={"ID":"7254fe79-25df-49c3-843c-51f10d54f5f4","Type":"ContainerStarted","Data":"b6df3116ab3f9f371132fe47129bbd7c5abcf33781819f98590e80fa762aa1da"} Oct 30 18:05:49.908169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.908154 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:49.948830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.948779 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-85779f77c6-shks7" podStartSLOduration=1.003305662 podStartE2EDuration="1.948766396s" totalImagesPullingTime="945.460734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:49.946066466 +0000 UTC m=+448.995111380" watchObservedRunningTime="2026-10-30 18:05:49.948766396 +0000 UTC m=+448.997811309" Oct 30 18:05:49.979333 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.979302 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h44fj"] Oct 30 18:05:49.979563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:49.979536 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" podUID="e6f7d7ef-6012-4fdc-865d-6a642f2ac992" containerName="authorino" containerID="cri-o://188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796" gracePeriod=30 Oct 30 18:05:50.006929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.006888 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-d4mxd"] Oct 30 18:05:50.013759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.013725 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-d4mxd"] Oct 30 18:05:50.051369 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.051316 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" podStartSLOduration=2.462832894 podStartE2EDuration="3.051297805s" totalImagesPullingTime="588.464911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:50.051050595 +0000 UTC m=+449.100095508" watchObservedRunningTime="2026-10-30 18:05:50.051297805 +0000 UTC m=+449.100342720" Oct 30 18:05:50.287924 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.287896 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" Oct 30 18:05:50.443995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.443904 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f7d7ef-6012-4fdc-865d-6a642f2ac992-kube-api-access-tl89z\" (UniqueName: \"kubernetes.io/projected/e6f7d7ef-6012-4fdc-865d-6a642f2ac992-kube-api-access-tl89z\") pod \"e6f7d7ef-6012-4fdc-865d-6a642f2ac992\" (UID: \"e6f7d7ef-6012-4fdc-865d-6a642f2ac992\") " Oct 30 18:05:50.446386 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.446351 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f7d7ef-6012-4fdc-865d-6a642f2ac992-kube-api-access-tl89z" pod "e6f7d7ef-6012-4fdc-865d-6a642f2ac992" (UID: "e6f7d7ef-6012-4fdc-865d-6a642f2ac992"). InnerVolumeSpecName "kube-api-access-tl89z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:50.544678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.544643 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl89z\" (UniqueName: \"kubernetes.io/projected/e6f7d7ef-6012-4fdc-865d-6a642f2ac992-kube-api-access-tl89z\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:50.920096 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.920053 4862 generic.go:356] "Generic (PLEG): container finished" podID="e6f7d7ef-6012-4fdc-865d-6a642f2ac992" containerID="188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796" exitCode=0 Oct 30 18:05:50.920588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.920545 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" event={"ID":"e6f7d7ef-6012-4fdc-865d-6a642f2ac992","Type":"ContainerDied","Data":"188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796"} Oct 30 18:05:50.920667 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.920590 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" event={"ID":"e6f7d7ef-6012-4fdc-865d-6a642f2ac992","Type":"ContainerDied","Data":"9baf805b33e4ac16729ef6673128713534e767f43245c244da53e5968c1be782"} Oct 30 18:05:50.920667 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.920627 4862 scope.go:122] "RemoveContainer" containerID="188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796" Oct 30 18:05:50.920747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.920679 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-h44fj" Oct 30 18:05:50.930206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.930189 4862 scope.go:122] "RemoveContainer" containerID="188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796" Oct 30 18:05:50.930514 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:05:50.930498 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796\": container with ID starting with 188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796 not found: ID does not exist" containerID="188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796" Oct 30 18:05:50.930565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.930523 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796"} err="failed to get container status \"188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796\": rpc error: code = NotFound desc = could not find container \"188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796\": container with ID starting with 188a6cae4b35041eee0545a75b81462db50c8c6fae6c1cbaaded9fbb0bac6796 not found: ID does not exist" Oct 30 18:05:50.966552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.966524 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h44fj"] Oct 30 18:05:50.968015 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:50.967994 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-h44fj"] Oct 30 18:05:51.060184 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:51.060157 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e87056-1721-40dc-95ba-f2166492c523" path="/var/lib/kubelet/pods/74e87056-1721-40dc-95ba-f2166492c523/volumes" Oct 30 18:05:51.060452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:51.060438 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f7d7ef-6012-4fdc-865d-6a642f2ac992" path="/var/lib/kubelet/pods/e6f7d7ef-6012-4fdc-865d-6a642f2ac992/volumes" Oct 30 18:05:55.925672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:55.925640 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b4d8998bc-4lr55" Oct 30 18:05:55.998804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:55.998775 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5"] Oct 30 18:05:55.999041 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:55.999017 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" podUID="e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" containerName="maas-api" containerID="cri-o://8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0" gracePeriod=30 Oct 30 18:05:56.242668 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.242646 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:56.288821 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.288792 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-metrics-certs\") pod \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " Oct 30 18:05:56.288961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.288875 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-openshift-service-ca\") pod \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " Oct 30 18:05:56.288961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.288926 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-kube-api-access-9f6mh\" (UniqueName: \"kubernetes.io/projected/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-kube-api-access-9f6mh\") pod \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " Oct 30 18:05:56.289074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.288969 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-maas-api-tls\") pod \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\" (UID: \"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0\") " Oct 30 18:05:56.289420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.289377 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-openshift-service-ca" pod "e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" (UID: "e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:56.291416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.291393 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-kube-api-access-9f6mh" pod "e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" (UID: "e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0"). InnerVolumeSpecName "kube-api-access-9f6mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:56.291614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.291540 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-maas-api-tls" pod "e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" (UID: "e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:56.291865 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.291849 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-metrics-certs" pod "e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" (UID: "e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:56.389549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.389520 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6mh\" (UniqueName: \"kubernetes.io/projected/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-kube-api-access-9f6mh\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:56.389549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.389545 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-maas-api-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:56.389549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.389556 4862 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-metrics-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:56.389739 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.389564 4862 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0-openshift-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:05:56.948144 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.948112 4862 generic.go:356] "Generic (PLEG): container finished" podID="e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" containerID="8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0" exitCode=0 Oct 30 18:05:56.948594 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.948167 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" Oct 30 18:05:56.948594 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.948209 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" event={"ID":"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0","Type":"ContainerDied","Data":"8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0"} Oct 30 18:05:56.948594 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.948278 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5" event={"ID":"e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0","Type":"ContainerDied","Data":"684fad1cd7de9bf450f02d5e8823cb0a9e74aade8fe8c603f0a4979a77dda711"} Oct 30 18:05:56.948594 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.948287 4862 scope.go:122] "RemoveContainer" containerID="8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0" Oct 30 18:05:56.957072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.957058 4862 scope.go:122] "RemoveContainer" containerID="8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0" Oct 30 18:05:56.957310 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:05:56.957287 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0\": container with ID starting with 8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0 not found: ID does not exist" containerID="8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0" Oct 30 18:05:56.957390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.957319 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0"} err="failed to get container status \"8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0\": rpc error: code = NotFound desc = could not find container \"8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0\": container with ID starting with 8e1167cebfae66465bbf313f6a78422f2d44803273e64c054dc8e2abcb9411d0 not found: ID does not exist" Oct 30 18:05:56.995912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:56.995883 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5"] Oct 30 18:05:57.006048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:57.006027 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65d69b8749-q2wd5"] Oct 30 18:05:57.059413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:05:57.059393 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" path="/var/lib/kubelet/pods/e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0/volumes" Oct 30 18:06:11.100224 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.100189 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr"] Oct 30 18:06:11.100643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.100572 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e6f7d7ef-6012-4fdc-865d-6a642f2ac992" containerName="authorino" Oct 30 18:06:11.100643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.100605 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f7d7ef-6012-4fdc-865d-6a642f2ac992" containerName="authorino" Oct 30 18:06:11.100643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.100635 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" containerName="maas-api" Oct 30 18:06:11.100643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.100643 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" containerName="maas-api" Oct 30 18:06:11.100843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.100743 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f7d7ef-6012-4fdc-865d-6a642f2ac992" containerName="authorino" Oct 30 18:06:11.100843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.100759 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e624d37e-68a2-4fb1-ad1f-5dc5c77db2f0" containerName="maas-api" Oct 30 18:06:11.107105 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.107088 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:11.126200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.126172 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr"] Oct 30 18:06:11.201726 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.201705 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58092d26-be96-4cc3-8584-cb9b729475dd-cert\") pod \"llmisvc-controller-manager-6466c554d8-jzvfr\" (UID: \"58092d26-be96-4cc3-8584-cb9b729475dd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:11.201836 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.201757 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ml4b\" (UniqueName: \"kubernetes.io/projected/58092d26-be96-4cc3-8584-cb9b729475dd-kube-api-access-2ml4b\") pod \"llmisvc-controller-manager-6466c554d8-jzvfr\" (UID: \"58092d26-be96-4cc3-8584-cb9b729475dd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:11.302986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.302958 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ml4b\" (UniqueName: \"kubernetes.io/projected/58092d26-be96-4cc3-8584-cb9b729475dd-kube-api-access-2ml4b\") pod \"llmisvc-controller-manager-6466c554d8-jzvfr\" (UID: \"58092d26-be96-4cc3-8584-cb9b729475dd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:11.303081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.303012 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58092d26-be96-4cc3-8584-cb9b729475dd-cert\") pod \"llmisvc-controller-manager-6466c554d8-jzvfr\" (UID: \"58092d26-be96-4cc3-8584-cb9b729475dd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:11.305526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.305504 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58092d26-be96-4cc3-8584-cb9b729475dd-cert\") pod \"llmisvc-controller-manager-6466c554d8-jzvfr\" (UID: \"58092d26-be96-4cc3-8584-cb9b729475dd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:11.327109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.327085 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ml4b\" (UniqueName: \"kubernetes.io/projected/58092d26-be96-4cc3-8584-cb9b729475dd-kube-api-access-2ml4b\") pod \"llmisvc-controller-manager-6466c554d8-jzvfr\" (UID: \"58092d26-be96-4cc3-8584-cb9b729475dd\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:11.419462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.419413 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:11.712864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:11.712711 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr"] Oct 30 18:06:11.718023 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:06:11.717989 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58092d26_be96_4cc3_8584_cb9b729475dd.slice/crio-34d3e21530a25a3b3e4fbdc100a8a190f3b9065035e795792ff0a2a0d558882b WatchSource:0}: Error finding container 34d3e21530a25a3b3e4fbdc100a8a190f3b9065035e795792ff0a2a0d558882b: Status 404 returned error can't find the container with id 34d3e21530a25a3b3e4fbdc100a8a190f3b9065035e795792ff0a2a0d558882b Oct 30 18:06:12.011558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:12.011487 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" event={"ID":"58092d26-be96-4cc3-8584-cb9b729475dd","Type":"ContainerStarted","Data":"34d3e21530a25a3b3e4fbdc100a8a190f3b9065035e795792ff0a2a0d558882b"} Oct 30 18:06:13.017156 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:13.017123 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" event={"ID":"58092d26-be96-4cc3-8584-cb9b729475dd","Type":"ContainerStarted","Data":"2d66e9706aa1ea0bf90ddbbcf034c575b50760bff701b351af348d0a982a8255"} Oct 30 18:06:13.017563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:13.017200 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:13.058340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:13.058298 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" podStartSLOduration=1.733838406 podStartE2EDuration="2.058284858s" totalImagesPullingTime="324.446452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:13.057681549 +0000 UTC m=+472.106726465" watchObservedRunningTime="2026-10-30 18:06:13.058284858 +0000 UTC m=+472.107329773" Oct 30 18:06:19.870558 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:06:19.870525 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:06:44.024324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.024230 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:06:44.092526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.092492 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95"] Oct 30 18:06:44.092753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.092725 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" podUID="49c5b13b-46cc-4116-a59d-ca7abeecb539" containerName="manager" containerID="cri-o://37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a" gracePeriod=30 Oct 30 18:06:44.334613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.334590 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:06:44.445888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.445855 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c5b13b-46cc-4116-a59d-ca7abeecb539-kube-api-access-rptht\" (UniqueName: \"kubernetes.io/projected/49c5b13b-46cc-4116-a59d-ca7abeecb539-kube-api-access-rptht\") pod \"49c5b13b-46cc-4116-a59d-ca7abeecb539\" (UID: \"49c5b13b-46cc-4116-a59d-ca7abeecb539\") " Oct 30 18:06:44.446062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.445950 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49c5b13b-46cc-4116-a59d-ca7abeecb539-cert\" (UniqueName: \"kubernetes.io/secret/49c5b13b-46cc-4116-a59d-ca7abeecb539-cert\") pod \"49c5b13b-46cc-4116-a59d-ca7abeecb539\" (UID: \"49c5b13b-46cc-4116-a59d-ca7abeecb539\") " Oct 30 18:06:44.448156 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.448129 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c5b13b-46cc-4116-a59d-ca7abeecb539-kube-api-access-rptht" pod "49c5b13b-46cc-4116-a59d-ca7abeecb539" (UID: "49c5b13b-46cc-4116-a59d-ca7abeecb539"). InnerVolumeSpecName "kube-api-access-rptht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:44.448286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.448182 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49c5b13b-46cc-4116-a59d-ca7abeecb539-cert" pod "49c5b13b-46cc-4116-a59d-ca7abeecb539" (UID: "49c5b13b-46cc-4116-a59d-ca7abeecb539"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:44.546879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.546853 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rptht\" (UniqueName: \"kubernetes.io/projected/49c5b13b-46cc-4116-a59d-ca7abeecb539-kube-api-access-rptht\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:06:44.546879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:44.546877 4862 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49c5b13b-46cc-4116-a59d-ca7abeecb539-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.146418 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:45.146383 4862 generic.go:356] "Generic (PLEG): container finished" podID="49c5b13b-46cc-4116-a59d-ca7abeecb539" containerID="37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a" exitCode=0 Oct 30 18:06:45.146841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:45.146450 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" Oct 30 18:06:45.146841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:45.146473 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" event={"ID":"49c5b13b-46cc-4116-a59d-ca7abeecb539","Type":"ContainerDied","Data":"37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a"} Oct 30 18:06:45.146841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:45.146552 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95" event={"ID":"49c5b13b-46cc-4116-a59d-ca7abeecb539","Type":"ContainerDied","Data":"309393e2e0f62e71f2efe8b12b15fac067d3d8c766b4ae0a9ad0e2254320f951"} Oct 30 18:06:45.146841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:45.146575 4862 scope.go:122] "RemoveContainer" containerID="37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a" Oct 30 18:06:45.154978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:45.154958 4862 scope.go:122] "RemoveContainer" containerID="37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a" Oct 30 18:06:45.155255 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:06:45.155237 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a\": container with ID starting with 37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a not found: ID does not exist" containerID="37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a" Oct 30 18:06:45.155318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:45.155280 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a"} err="failed to get container status \"37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a\": rpc error: code = NotFound desc = could not find container \"37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a\": container with ID starting with 37ad04e29af3f3d13355481aa81d76933e59d79394627ad5404f0dbfa151263a not found: ID does not exist" Oct 30 18:06:45.195888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:45.195870 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95"] Oct 30 18:06:45.206106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:45.206036 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nwq95"] Oct 30 18:06:47.060063 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:06:47.060029 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c5b13b-46cc-4116-a59d-ca7abeecb539" path="/var/lib/kubelet/pods/49c5b13b-46cc-4116-a59d-ca7abeecb539/volumes" Oct 30 18:06:49.868037 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:06:49.868003 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:07:19.870455 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:07:19.870417 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:07:20.767294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:20.766350 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:07:23.995509 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:23.995472 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:07:28.997906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:28.997869 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:07:32.919412 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:32.919373 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:07:35.863524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.863472 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn"] Oct 30 18:07:35.864725 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.864700 4862 cpu_manager.go:414] "containerMap: removing container" podUID="49c5b13b-46cc-4116-a59d-ca7abeecb539" containerName="manager" Oct 30 18:07:35.864892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.864869 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c5b13b-46cc-4116-a59d-ca7abeecb539" containerName="manager" Oct 30 18:07:35.865047 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.864987 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c5b13b-46cc-4116-a59d-ca7abeecb539" containerName="manager" Oct 30 18:07:35.869525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.869505 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:35.886470 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.886451 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:07:35.886584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.886448 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:07:35.886584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.886540 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 18:07:35.898812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.898790 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hdx68\"" Oct 30 18:07:35.900200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.900174 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn"] Oct 30 18:07:35.954319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.954296 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:35.954472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.954336 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:35.954472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.954356 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da01fa56-122a-490d-999b-e2c20b957e4e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:35.954472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.954378 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8v25\" (UniqueName: \"kubernetes.io/projected/da01fa56-122a-490d-999b-e2c20b957e4e-kube-api-access-j8v25\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:35.954576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.954512 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:35.954576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:35.954565 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.054951 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.054916 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8v25\" (UniqueName: \"kubernetes.io/projected/da01fa56-122a-490d-999b-e2c20b957e4e-kube-api-access-j8v25\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.054951 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.054953 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.055205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.054991 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.055205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.055037 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.055205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.055085 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.055205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.055108 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da01fa56-122a-490d-999b-e2c20b957e4e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.055462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.055389 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.055518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.055503 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.055570 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.055553 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.057738 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.057715 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da01fa56-122a-490d-999b-e2c20b957e4e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.057934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.057912 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da01fa56-122a-490d-999b-e2c20b957e4e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.078309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.078280 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8v25\" (UniqueName: \"kubernetes.io/projected/da01fa56-122a-490d-999b-e2c20b957e4e-kube-api-access-j8v25\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn\" (UID: \"da01fa56-122a-490d-999b-e2c20b957e4e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.184521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.184437 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:36.486736 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:36.486664 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn"] Oct 30 18:07:36.489860 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:07:36.489824 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda01fa56_122a_490d_999b_e2c20b957e4e.slice/crio-832593acd33e0923acc5b209db137cb53eddbd76a8ff026fae84a32a309c8645 WatchSource:0}: Error finding container 832593acd33e0923acc5b209db137cb53eddbd76a8ff026fae84a32a309c8645: Status 404 returned error can't find the container with id 832593acd33e0923acc5b209db137cb53eddbd76a8ff026fae84a32a309c8645 Oct 30 18:07:37.369573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:37.369516 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" event={"ID":"da01fa56-122a-490d-999b-e2c20b957e4e","Type":"ContainerStarted","Data":"832593acd33e0923acc5b209db137cb53eddbd76a8ff026fae84a32a309c8645"} Oct 30 18:07:38.375379 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:38.375345 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" event={"ID":"da01fa56-122a-490d-999b-e2c20b957e4e","Type":"ContainerStarted","Data":"3a6dc66680b8cfdd1c2ca49c492e3d0ccfad196aa0792ab2e8e42fbaaeadee55"} Oct 30 18:07:38.418663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:38.418617 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podStartSLOduration=2.125681647 podStartE2EDuration="3.418604731s" totalImagesPullingTime="1.292923084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:38.417386304 +0000 UTC m=+557.466431213" watchObservedRunningTime="2026-10-30 18:07:38.418604731 +0000 UTC m=+557.467649634" Oct 30 18:07:39.113203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:39.113165 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:07:39.184887 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:39.184858 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:46.185515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:46.185484 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:46.197811 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:46.197788 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:46.422141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:46.422111 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:07:49.868214 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:07:49.868177 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:07:49.972846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:49.972749 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg"] Oct 30 18:07:49.978347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:49.977729 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.004480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.004445 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg"] Oct 30 18:07:50.076592 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.076561 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:07:50.076592 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.076564 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xfzjq\"" Oct 30 18:07:50.081929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.081905 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.082057 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.081948 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.082057 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.082035 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsw2l\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-kube-api-access-gsw2l\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.082148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.082097 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.082182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.082157 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.082214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.082191 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.082214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.082207 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.082334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.082320 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.082371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.082357 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.182945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.182873 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.182945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.182909 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.182945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.182925 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsw2l\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-kube-api-access-gsw2l\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.182945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.182946 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.182968 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.182986 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.183004 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.183041 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.183107 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.183417 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.183485 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183653 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.183563 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.183653 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.183770 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.183750 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.185957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.185936 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.186203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.186186 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.202542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.202517 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsw2l\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-kube-api-access-gsw2l\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.202633 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.202581 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-6qqzg\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.290271 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.290246 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:50.602337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:50.602312 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg"] Oct 30 18:07:50.603486 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:07:50.603463 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5b64a1_2f6b_4120_ab40_76b3bc209992.slice/crio-3f9a055ef82536a9acfb99ba2bfd195e0790628c8107285e81a65bbf12dac6a4 WatchSource:0}: Error finding container 3f9a055ef82536a9acfb99ba2bfd195e0790628c8107285e81a65bbf12dac6a4: Status 404 returned error can't find the container with id 3f9a055ef82536a9acfb99ba2bfd195e0790628c8107285e81a65bbf12dac6a4 Oct 30 18:07:51.440743 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.440701 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" event={"ID":"3a5b64a1-2f6b-4120-ab40-76b3bc209992","Type":"ContainerStarted","Data":"3f9a055ef82536a9acfb99ba2bfd195e0790628c8107285e81a65bbf12dac6a4"} Oct 30 18:07:51.503049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.503017 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5748484d8c-wtztm"] Oct 30 18:07:51.506588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.506556 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5748484d8c-wtztm" Oct 30 18:07:51.526652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.526621 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5748484d8c-wtztm"] Oct 30 18:07:51.594795 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.594759 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4zj\" (UniqueName: \"kubernetes.io/projected/8db3b37b-c28b-4b43-8e5d-7e285f8a6819-kube-api-access-dg4zj\") pod \"authorino-5748484d8c-wtztm\" (UID: \"8db3b37b-c28b-4b43-8e5d-7e285f8a6819\") " pod="kuadrant-system/authorino-5748484d8c-wtztm" Oct 30 18:07:51.594976 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.594831 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8db3b37b-c28b-4b43-8e5d-7e285f8a6819-tls-cert\") pod \"authorino-5748484d8c-wtztm\" (UID: \"8db3b37b-c28b-4b43-8e5d-7e285f8a6819\") " pod="kuadrant-system/authorino-5748484d8c-wtztm" Oct 30 18:07:51.696381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.696303 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8db3b37b-c28b-4b43-8e5d-7e285f8a6819-tls-cert\") pod \"authorino-5748484d8c-wtztm\" (UID: \"8db3b37b-c28b-4b43-8e5d-7e285f8a6819\") " pod="kuadrant-system/authorino-5748484d8c-wtztm" Oct 30 18:07:51.696535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.696405 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4zj\" (UniqueName: \"kubernetes.io/projected/8db3b37b-c28b-4b43-8e5d-7e285f8a6819-kube-api-access-dg4zj\") pod \"authorino-5748484d8c-wtztm\" (UID: \"8db3b37b-c28b-4b43-8e5d-7e285f8a6819\") " pod="kuadrant-system/authorino-5748484d8c-wtztm" Oct 30 18:07:51.699208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.699186 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8db3b37b-c28b-4b43-8e5d-7e285f8a6819-tls-cert\") pod \"authorino-5748484d8c-wtztm\" (UID: \"8db3b37b-c28b-4b43-8e5d-7e285f8a6819\") " pod="kuadrant-system/authorino-5748484d8c-wtztm" Oct 30 18:07:51.716075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.716049 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4zj\" (UniqueName: \"kubernetes.io/projected/8db3b37b-c28b-4b43-8e5d-7e285f8a6819-kube-api-access-dg4zj\") pod \"authorino-5748484d8c-wtztm\" (UID: \"8db3b37b-c28b-4b43-8e5d-7e285f8a6819\") " pod="kuadrant-system/authorino-5748484d8c-wtztm" Oct 30 18:07:51.820981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:51.820948 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5748484d8c-wtztm" Oct 30 18:07:52.123596 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:52.123568 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5748484d8c-wtztm"] Oct 30 18:07:52.127056 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:07:52.127023 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8db3b37b_c28b_4b43_8e5d_7e285f8a6819.slice/crio-cc8ad54cb07af8a2307269fef7d2a99ba1ca5f10e7e878a196fad15361fbdbb1 WatchSource:0}: Error finding container cc8ad54cb07af8a2307269fef7d2a99ba1ca5f10e7e878a196fad15361fbdbb1: Status 404 returned error can't find the container with id cc8ad54cb07af8a2307269fef7d2a99ba1ca5f10e7e878a196fad15361fbdbb1 Oct 30 18:07:52.448443 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:52.448374 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5748484d8c-wtztm" event={"ID":"8db3b37b-c28b-4b43-8e5d-7e285f8a6819","Type":"ContainerStarted","Data":"cc8ad54cb07af8a2307269fef7d2a99ba1ca5f10e7e878a196fad15361fbdbb1"} Oct 30 18:07:53.243345 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.243306 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:53.243476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.243398 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:53.243476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.243444 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:07:53.458285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.458004 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5748484d8c-wtztm" event={"ID":"8db3b37b-c28b-4b43-8e5d-7e285f8a6819","Type":"ContainerStarted","Data":"e92ba2937c33c67b7272db38c1cf695238eaa341335ad2e20ad1baabd703fdc1"} Oct 30 18:07:53.460047 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.460014 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" event={"ID":"3a5b64a1-2f6b-4120-ab40-76b3bc209992","Type":"ContainerStarted","Data":"e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56"} Oct 30 18:07:53.505353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.505013 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5748484d8c-wtztm" podStartSLOduration=1.685145172 podStartE2EDuration="2.504996926s" totalImagesPullingTime="819.851754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:53.504727385 +0000 UTC m=+572.553772300" watchObservedRunningTime="2026-10-30 18:07:53.504996926 +0000 UTC m=+572.554041839" Oct 30 18:07:53.536077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.536029 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85779f77c6-shks7"] Oct 30 18:07:53.536568 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.536533 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-85779f77c6-shks7" podUID="5c67be0f-89db-4e9c-a225-21f6bd526718" containerName="authorino" containerID="cri-o://0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954" gracePeriod=30 Oct 30 18:07:53.556543 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.556487 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" podStartSLOduration=1.919468159 podStartE2EDuration="4.556469058s" totalImagesPullingTime="2.637000899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:53.554524489 +0000 UTC m=+572.603569403" watchObservedRunningTime="2026-10-30 18:07:53.556469058 +0000 UTC m=+572.605513971" Oct 30 18:07:53.862636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:53.862598 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:07:54.018032 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.018002 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c67be0f-89db-4e9c-a225-21f6bd526718-kube-api-access-4whv6\" (UniqueName: \"kubernetes.io/projected/5c67be0f-89db-4e9c-a225-21f6bd526718-kube-api-access-4whv6\") pod \"5c67be0f-89db-4e9c-a225-21f6bd526718\" (UID: \"5c67be0f-89db-4e9c-a225-21f6bd526718\") " Oct 30 18:07:54.018210 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.018063 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c67be0f-89db-4e9c-a225-21f6bd526718-tls-cert\" (UniqueName: \"kubernetes.io/secret/5c67be0f-89db-4e9c-a225-21f6bd526718-tls-cert\") pod \"5c67be0f-89db-4e9c-a225-21f6bd526718\" (UID: \"5c67be0f-89db-4e9c-a225-21f6bd526718\") " Oct 30 18:07:54.020329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.020298 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c67be0f-89db-4e9c-a225-21f6bd526718-kube-api-access-4whv6" pod "5c67be0f-89db-4e9c-a225-21f6bd526718" (UID: "5c67be0f-89db-4e9c-a225-21f6bd526718"). InnerVolumeSpecName "kube-api-access-4whv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:54.044469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.044403 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c67be0f-89db-4e9c-a225-21f6bd526718-tls-cert" pod "5c67be0f-89db-4e9c-a225-21f6bd526718" (UID: "5c67be0f-89db-4e9c-a225-21f6bd526718"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:54.119208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.119125 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whv6\" (UniqueName: \"kubernetes.io/projected/5c67be0f-89db-4e9c-a225-21f6bd526718-kube-api-access-4whv6\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:07:54.119208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.119158 4862 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c67be0f-89db-4e9c-a225-21f6bd526718-tls-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:07:54.290402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.290368 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:07:54.291924 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.291898 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-6qqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:07:54.292066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.291954 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:07:54.465899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.465809 4862 generic.go:356] "Generic (PLEG): container finished" podID="5c67be0f-89db-4e9c-a225-21f6bd526718" containerID="0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954" exitCode=0 Oct 30 18:07:54.465899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.465840 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85779f77c6-shks7" Oct 30 18:07:54.465899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.465882 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85779f77c6-shks7" event={"ID":"5c67be0f-89db-4e9c-a225-21f6bd526718","Type":"ContainerDied","Data":"0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954"} Oct 30 18:07:54.466451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.465919 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85779f77c6-shks7" event={"ID":"5c67be0f-89db-4e9c-a225-21f6bd526718","Type":"ContainerDied","Data":"9416adaf0e9293b363cf539e6356080e6605aaa36f086fe0940565426c11b7dd"} Oct 30 18:07:54.466451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.465934 4862 scope.go:122] "RemoveContainer" containerID="0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954" Oct 30 18:07:54.476843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.476823 4862 scope.go:122] "RemoveContainer" containerID="0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954" Oct 30 18:07:54.477125 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:07:54.477104 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954\": container with ID starting with 0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954 not found: ID does not exist" containerID="0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954" Oct 30 18:07:54.477203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.477135 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954"} err="failed to get container status \"0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954\": rpc error: code = NotFound desc = could not find container \"0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954\": container with ID starting with 0c06ce07287ec928e2610700f5c348b65779c3e94c1ff1c99a1cf369e7906954 not found: ID does not exist" Oct 30 18:07:54.519679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.519648 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85779f77c6-shks7"] Oct 30 18:07:54.523969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:54.523943 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85779f77c6-shks7"] Oct 30 18:07:55.062052 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:55.062022 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c67be0f-89db-4e9c-a225-21f6bd526718" path="/var/lib/kubelet/pods/5c67be0f-89db-4e9c-a225-21f6bd526718/volumes" Oct 30 18:07:55.290831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:55.290797 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-6qqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:07:55.290985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:55.290864 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:07:56.291419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:56.291386 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-6qqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:07:56.291801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:56.291435 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:07:57.290747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:57.290714 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-6qqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:07:57.290926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:57.290766 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:07:58.291276 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:58.291230 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-6qqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:07:58.291687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:58.291304 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:07:59.291684 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:59.291644 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-6qqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:07:59.292098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:07:59.291711 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:08:00.291290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:00.291223 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-6qqzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 18:08:00.291490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:00.291313 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 18:08:01.690023 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:01.689662 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:08:01.690496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:01.690176 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:08:01.691332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:01.691314 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:08:05.033388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.033353 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs"] Oct 30 18:08:05.033856 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.033843 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5c67be0f-89db-4e9c-a225-21f6bd526718" containerName="authorino" Oct 30 18:08:05.033925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.033861 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c67be0f-89db-4e9c-a225-21f6bd526718" containerName="authorino" Oct 30 18:08:05.033980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.033972 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c67be0f-89db-4e9c-a225-21f6bd526718" containerName="authorino" Oct 30 18:08:05.037039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.037018 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.061876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.061851 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs"] Oct 30 18:08:05.143216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.143191 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.143359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.143224 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.143359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.143247 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.143359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.143345 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.143500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.143363 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvp72\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-kube-api-access-bvp72\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.143500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.143382 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64350f03-8872-451e-80ed-2f2a7b837c55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.143500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.143459 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.143500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.143491 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.143630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.143507 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64350f03-8872-451e-80ed-2f2a7b837c55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244359 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244387 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvp72\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-kube-api-access-bvp72\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244405 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64350f03-8872-451e-80ed-2f2a7b837c55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244436 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244546 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244576 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64350f03-8872-451e-80ed-2f2a7b837c55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244683 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244628 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244683 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244665 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244786 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244705 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244788 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244873 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.244966 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.244905 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.245043 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.245018 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.245389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.245367 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64350f03-8872-451e-80ed-2f2a7b837c55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.246944 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.246925 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.247170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.247153 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64350f03-8872-451e-80ed-2f2a7b837c55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.263768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.263750 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvp72\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-kube-api-access-bvp72\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.263847 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.263815 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-w8sfs\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.350365 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.350342 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:05.662115 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.662092 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs"] Oct 30 18:08:05.663781 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:08:05.663734 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64350f03_8872_451e_80ed_2f2a7b837c55.slice/crio-a6ca121a0971c33961624cf2fc24e5cee694c38a8bbd4167e27041eb6ffaea0d WatchSource:0}: Error finding container a6ca121a0971c33961624cf2fc24e5cee694c38a8bbd4167e27041eb6ffaea0d: Status 404 returned error can't find the container with id a6ca121a0971c33961624cf2fc24e5cee694c38a8bbd4167e27041eb6ffaea0d Oct 30 18:08:05.666760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.666731 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:05.666836 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.666802 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:05.666870 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:05.666833 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:06.528164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:06.528126 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" event={"ID":"64350f03-8872-451e-80ed-2f2a7b837c55","Type":"ContainerStarted","Data":"60f8c510e1335eca79e9b90e3f7abded45db2f44d5df41803245b53f4bdf91f5"} Oct 30 18:08:06.528671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:06.528180 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" event={"ID":"64350f03-8872-451e-80ed-2f2a7b837c55","Type":"ContainerStarted","Data":"a6ca121a0971c33961624cf2fc24e5cee694c38a8bbd4167e27041eb6ffaea0d"} Oct 30 18:08:07.353290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:07.352569 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:07.354801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:07.354777 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w8sfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:07.354921 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:07.354831 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 18:08:08.353674 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:08.353639 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w8sfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:08.354069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:08.353694 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 18:08:09.353739 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:09.353704 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w8sfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:09.354117 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:09.353782 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 18:08:10.353624 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:10.353589 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w8sfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:10.353828 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:10.353661 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 18:08:11.353751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:11.353714 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w8sfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:11.353953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:11.353791 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 18:08:12.351453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:12.351421 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w8sfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 18:08:12.351613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:12.351476 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 18:08:13.809951 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:13.809923 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:13.810414 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:13.810369 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:13.811154 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:13.811130 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:08:13.867355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:13.867255 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" podStartSLOduration=8.867242574 podStartE2EDuration="8.867242574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:06.601411388 +0000 UTC m=+585.650456283" watchObservedRunningTime="2026-10-30 18:08:13.867242574 +0000 UTC m=+592.916287486" Oct 30 18:08:19.870169 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:08:19.870125 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:08:20.054827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.054796 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6"] Oct 30 18:08:20.059433 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.059413 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.085767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.085746 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6"] Oct 30 18:08:20.188959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.188883 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.188959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.188926 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.189179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.189162 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6jp\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-kube-api-access-6v6jp\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.189232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.189187 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.189232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.189212 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.189375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.189249 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.189375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.189356 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.189460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.189386 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.189460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.189405 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290425 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290402 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290429 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290449 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290542 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290585 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290642 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290681 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290776 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290780 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6jp\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-kube-api-access-6v6jp\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290824 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.290876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290826 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.291067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.290944 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.291193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.291165 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.291370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.291353 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.293030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.293012 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.293172 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.293156 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.311058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.311030 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.311211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.311193 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6jp\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-kube-api-access-6v6jp\") pod \"maas-default-gateway-openshift-default-76597d6855-wgrr6\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.372248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.372225 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:20.689029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.689002 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6"] Oct 30 18:08:20.690969 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:08:20.690941 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fc7558_2c7b_45af_ad4f_4091b840ee5c.slice/crio-e8defdfcf80f1613df9ea11af02f554dfaefc2d95b2e320d096268b8560f9c5b WatchSource:0}: Error finding container e8defdfcf80f1613df9ea11af02f554dfaefc2d95b2e320d096268b8560f9c5b: Status 404 returned error can't find the container with id e8defdfcf80f1613df9ea11af02f554dfaefc2d95b2e320d096268b8560f9c5b Oct 30 18:08:20.693840 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.693812 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:20.693912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.693879 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:20.693952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:20.693914 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:08:21.104766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:21.104733 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:08:21.105867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:21.105844 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:08:21.618978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:21.618832 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" event={"ID":"42fc7558-2c7b-45af-ad4f-4091b840ee5c","Type":"ContainerStarted","Data":"b3fbfc3714bfc125ad50125030b8476b74aef303e9345a193782afb69b8423b7"} Oct 30 18:08:21.618978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:21.618901 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" event={"ID":"42fc7558-2c7b-45af-ad4f-4091b840ee5c","Type":"ContainerStarted","Data":"e8defdfcf80f1613df9ea11af02f554dfaefc2d95b2e320d096268b8560f9c5b"} Oct 30 18:08:21.706917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:21.706873 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" podStartSLOduration=1.706862014 podStartE2EDuration="1.706862014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:21.694505397 +0000 UTC m=+600.743550310" watchObservedRunningTime="2026-10-30 18:08:21.706862014 +0000 UTC m=+600.755906927" Oct 30 18:08:22.373310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:22.373280 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:22.374606 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:22.374580 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wgrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:22.374751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:22.374632 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:08:23.372580 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:23.372548 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wgrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:23.372755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:23.372599 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:08:24.372809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:24.372775 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wgrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:24.373197 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:24.372839 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:08:25.373467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:25.373439 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wgrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:25.373916 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:25.373890 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:08:26.372982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:26.372949 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wgrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:26.373171 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:26.373006 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:08:27.372695 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:27.372663 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wgrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 30 18:08:27.373065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:27.372710 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 30 18:08:28.936445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:28.936414 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:28.937064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:28.937046 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:28.937688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:08:28.937666 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:08:49.868341 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:08:49.868302 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:09:11.649930 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.649893 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h"] Oct 30 18:09:11.654313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.654289 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.677422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.677391 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h"] Oct 30 18:09:11.686751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.686728 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-57bf5177-gw-openshift-default-dockercfg-4k7s6\"" Oct 30 18:09:11.747700 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.747670 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd568017-d361-47cc-b9c3-9a0425d72ba8-istiod-ca-cert\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.747853 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.747720 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-socket\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.747907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.747850 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-token\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.747907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.747887 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkmn\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-kube-api-access-fdkmn\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.748003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.747929 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-envoy\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.748051 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.748020 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-data\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.748090 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.748068 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-podinfo\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.748137 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.748112 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-credential-socket\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.748181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.748168 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-certs\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.848766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.848734 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd568017-d361-47cc-b9c3-9a0425d72ba8-istiod-ca-cert\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.848946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.848793 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-socket\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.848946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.848835 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-token\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.848946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.848859 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkmn\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-kube-api-access-fdkmn\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.848946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.848897 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-envoy\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.848946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.848922 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-data\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.848946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.848946 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-podinfo\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.849312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.848981 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-credential-socket\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.849312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.849035 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-certs\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.849312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.849216 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-socket\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.849495 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.849466 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-certs\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.849553 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.849471 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-data\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.849635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.849611 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd568017-d361-47cc-b9c3-9a0425d72ba8-istiod-ca-cert\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.849755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.849728 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-credential-socket\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.852144 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.852109 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-podinfo\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.852144 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.852108 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-envoy\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.875186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.875154 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkmn\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-kube-api-access-fdkmn\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.879358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.879335 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-token\") pod \"e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:11.968657 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:11.968581 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:12.286908 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:12.286839 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h"] Oct 30 18:09:12.289051 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:09:12.289024 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd568017_d361_47cc_b9c3_9a0425d72ba8.slice/crio-094e08e2b86df03c2a95784af0f7923cf97c5d3e337840e5f3704a444781994b WatchSource:0}: Error finding container 094e08e2b86df03c2a95784af0f7923cf97c5d3e337840e5f3704a444781994b: Status 404 returned error can't find the container with id 094e08e2b86df03c2a95784af0f7923cf97c5d3e337840e5f3704a444781994b Oct 30 18:09:12.291139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:12.291106 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:12.291238 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:12.291179 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:12.291238 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:12.291219 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:12.864663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:12.864628 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" event={"ID":"cd568017-d361-47cc-b9c3-9a0425d72ba8","Type":"ContainerStarted","Data":"046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad"} Oct 30 18:09:12.865060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:12.864674 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" event={"ID":"cd568017-d361-47cc-b9c3-9a0425d72ba8","Type":"ContainerStarted","Data":"094e08e2b86df03c2a95784af0f7923cf97c5d3e337840e5f3704a444781994b"} Oct 30 18:09:12.922243 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:12.922197 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" podStartSLOduration=1.922178584 podStartE2EDuration="1.922178584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:12.920880684 +0000 UTC m=+651.969925599" watchObservedRunningTime="2026-10-30 18:09:12.922178584 +0000 UTC m=+651.971223496" Oct 30 18:09:12.969514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:12.969477 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:13.975003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:13.974913 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:14.873019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:14.872992 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:14.873955 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:14.873938 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:18.800353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:18.800313 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:09:19.407157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.407122 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm"] Oct 30 18:09:19.411035 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.411018 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:19.455980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.455933 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a521cf49\"" Oct 30 18:09:19.457685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.457658 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm"] Oct 30 18:09:19.522529 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.522500 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxcmq\" (UniqueName: \"kubernetes.io/projected/af4c61c9-855f-4083-aa0b-455fd6cdc719-kube-api-access-mxcmq\") pod \"payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm\" (UID: \"af4c61c9-855f-4083-aa0b-455fd6cdc719\") " pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:19.522678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.522591 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4c61c9-855f-4083-aa0b-455fd6cdc719-plugins-config-volume\") pod \"payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm\" (UID: \"af4c61c9-855f-4083-aa0b-455fd6cdc719\") " pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:19.624021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.623986 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4c61c9-855f-4083-aa0b-455fd6cdc719-plugins-config-volume\") pod \"payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm\" (UID: \"af4c61c9-855f-4083-aa0b-455fd6cdc719\") " pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:19.624204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.624065 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxcmq\" (UniqueName: \"kubernetes.io/projected/af4c61c9-855f-4083-aa0b-455fd6cdc719-kube-api-access-mxcmq\") pod \"payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm\" (UID: \"af4c61c9-855f-4083-aa0b-455fd6cdc719\") " pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:19.624686 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.624667 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4c61c9-855f-4083-aa0b-455fd6cdc719-plugins-config-volume\") pod \"payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm\" (UID: \"af4c61c9-855f-4083-aa0b-455fd6cdc719\") " pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:19.656110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.656073 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxcmq\" (UniqueName: \"kubernetes.io/projected/af4c61c9-855f-4083-aa0b-455fd6cdc719-kube-api-access-mxcmq\") pod \"payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm\" (UID: \"af4c61c9-855f-4083-aa0b-455fd6cdc719\") " pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:19.724751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:19.724669 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:19.886649 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:09:19.886607 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:09:20.050342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:20.049152 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm"] Oct 30 18:09:20.057086 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:09:20.057045 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4c61c9_855f_4083_aa0b_455fd6cdc719.slice/crio-72518f616aea444fb99fb713a1f77d3d5407f2b32aef2f1f01aa8110e517c2d7 WatchSource:0}: Error finding container 72518f616aea444fb99fb713a1f77d3d5407f2b32aef2f1f01aa8110e517c2d7: Status 404 returned error can't find the container with id 72518f616aea444fb99fb713a1f77d3d5407f2b32aef2f1f01aa8110e517c2d7 Oct 30 18:09:20.059358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:20.059335 4862 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:09:20.903794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:20.903763 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" event={"ID":"af4c61c9-855f-4083-aa0b-455fd6cdc719","Type":"ContainerStarted","Data":"72518f616aea444fb99fb713a1f77d3d5407f2b32aef2f1f01aa8110e517c2d7"} Oct 30 18:09:21.158503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.158425 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm"] Oct 30 18:09:21.163058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.163033 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.168463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.168234 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm"] Oct 30 18:09:21.179781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.179758 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a521cf49-serving-cert\"" Oct 30 18:09:21.260006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.259974 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-metrics-certs\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.260186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.260013 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrjn\" (UniqueName: \"kubernetes.io/projected/099b5d37-50b2-4704-9113-208a7361c6f7-kube-api-access-8zrjn\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.260186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.260081 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/099b5d37-50b2-4704-9113-208a7361c6f7-openshift-service-ca\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.260186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.260105 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-maas-api-tls\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.360917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.360889 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/099b5d37-50b2-4704-9113-208a7361c6f7-openshift-service-ca\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.360917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.360922 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-maas-api-tls\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.361119 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.360985 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-metrics-certs\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.361119 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.361002 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrjn\" (UniqueName: \"kubernetes.io/projected/099b5d37-50b2-4704-9113-208a7361c6f7-kube-api-access-8zrjn\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.362019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.361927 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/099b5d37-50b2-4704-9113-208a7361c6f7-openshift-service-ca\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.366045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.366021 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-metrics-certs\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.367203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.367184 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-maas-api-tls\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.386753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.385752 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrjn\" (UniqueName: \"kubernetes.io/projected/099b5d37-50b2-4704-9113-208a7361c6f7-kube-api-access-8zrjn\") pod \"maas-api-e2e-ait-a521cf49-59c877df55-nt8jm\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.483998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.483928 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:21.812536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.812506 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm"] Oct 30 18:09:21.816950 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:09:21.816921 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099b5d37_50b2_4704_9113_208a7361c6f7.slice/crio-30a6bbc5e183bf5adba405896ba4ec99a18cf309376f3a9d5b6c84a6497a6144 WatchSource:0}: Error finding container 30a6bbc5e183bf5adba405896ba4ec99a18cf309376f3a9d5b6c84a6497a6144: Status 404 returned error can't find the container with id 30a6bbc5e183bf5adba405896ba4ec99a18cf309376f3a9d5b6c84a6497a6144 Oct 30 18:09:21.913318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:21.910473 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" event={"ID":"099b5d37-50b2-4704-9113-208a7361c6f7","Type":"ContainerStarted","Data":"30a6bbc5e183bf5adba405896ba4ec99a18cf309376f3a9d5b6c84a6497a6144"} Oct 30 18:09:22.919768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:22.919740 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" event={"ID":"099b5d37-50b2-4704-9113-208a7361c6f7","Type":"ContainerStarted","Data":"9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa"} Oct 30 18:09:22.920384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:22.920359 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:22.975151 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:22.975099 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" podStartSLOduration=1.4928576599999999 podStartE2EDuration="1.975086854s" totalImagesPullingTime="482.229194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:22.974687617 +0000 UTC m=+662.023732527" watchObservedRunningTime="2026-10-30 18:09:22.975086854 +0000 UTC m=+662.024131767" Oct 30 18:09:23.945288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:23.934057 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" event={"ID":"af4c61c9-855f-4083-aa0b-455fd6cdc719","Type":"ContainerStarted","Data":"3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b"} Oct 30 18:09:23.945288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:23.934145 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:23.977846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:23.977751 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" podStartSLOduration=2.21539567 podStartE2EDuration="4.977737376s" totalImagesPullingTime="2.762341706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:23.976446615 +0000 UTC m=+663.025491564" watchObservedRunningTime="2026-10-30 18:09:23.977737376 +0000 UTC m=+663.026782289" Oct 30 18:09:24.617285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.615731 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2"] Oct 30 18:09:24.619965 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.619941 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.644287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.644207 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-594f46d3-serving-cert\"" Oct 30 18:09:24.663287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.662246 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2"] Oct 30 18:09:24.793632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.793584 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-openshift-service-ca\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.793632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.793635 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlps\" (UniqueName: \"kubernetes.io/projected/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-kube-api-access-pqlps\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.793949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.793764 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-maas-api-tls\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.793949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.793789 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-metrics-certs\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.895256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.895220 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-maas-api-tls\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.895420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.895312 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-metrics-certs\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.895462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.895416 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-openshift-service-ca\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.895462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.895442 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlps\" (UniqueName: \"kubernetes.io/projected/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-kube-api-access-pqlps\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.896183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.896156 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-openshift-service-ca\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.898436 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.898412 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-maas-api-tls\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.898436 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.898426 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-metrics-certs\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.915209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.915186 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlps\" (UniqueName: \"kubernetes.io/projected/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-kube-api-access-pqlps\") pod \"maas-api-e2e-ait-594f46d3-5889c999df-78zz2\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:24.941961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:24.941932 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:25.252117 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:25.252051 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2"] Oct 30 18:09:25.256029 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:09:25.256002 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f89a7c6_e6be_47a7_b5d1_35d08647bb30.slice/crio-47a449b0ed19bd4b2f5f54888372a56c73bb5b8493949770c42bf3b2dd3c1900 WatchSource:0}: Error finding container 47a449b0ed19bd4b2f5f54888372a56c73bb5b8493949770c42bf3b2dd3c1900: Status 404 returned error can't find the container with id 47a449b0ed19bd4b2f5f54888372a56c73bb5b8493949770c42bf3b2dd3c1900 Oct 30 18:09:25.945100 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:25.945004 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" event={"ID":"2f89a7c6-e6be-47a7-b5d1-35d08647bb30","Type":"ContainerStarted","Data":"277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274"} Oct 30 18:09:25.945100 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:25.945071 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" event={"ID":"2f89a7c6-e6be-47a7-b5d1-35d08647bb30","Type":"ContainerStarted","Data":"47a449b0ed19bd4b2f5f54888372a56c73bb5b8493949770c42bf3b2dd3c1900"} Oct 30 18:09:25.945358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:25.945253 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:25.988838 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:25.988777 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" podStartSLOduration=1.6319423130000001 podStartE2EDuration="1.988756383s" totalImagesPullingTime="356.81407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:25.985807403 +0000 UTC m=+665.034852400" watchObservedRunningTime="2026-10-30 18:09:25.988756383 +0000 UTC m=+665.037801296" Oct 30 18:09:28.606187 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:28.606146 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:09:28.962814 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:28.962719 4862 generic.go:356] "Generic (PLEG): container finished" podID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerID="046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad" exitCode=0 Oct 30 18:09:28.962986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:28.962799 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" event={"ID":"cd568017-d361-47cc-b9c3-9a0425d72ba8","Type":"ContainerDied","Data":"046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad"} Oct 30 18:09:28.963247 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:28.963229 4862 scope.go:122] "RemoveContainer" containerID="046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad" Oct 30 18:09:28.964548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:28.964517 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:28.964642 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:28.964583 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:28.964642 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:28.964639 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:29.873156 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:29.873129 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:29.955025 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:29.954997 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:29.969667 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:29.969638 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" event={"ID":"cd568017-d361-47cc-b9c3-9a0425d72ba8","Type":"ContainerStarted","Data":"053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f"} Oct 30 18:09:30.969168 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:30.969133 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:30.970477 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:30.970444 4862 patch_prober.go:29] interesting pod/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:09:30.970621 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:30.970513 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:09:31.958197 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:31.958164 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:31.969107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:31.969080 4862 patch_prober.go:29] interesting pod/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:09:31.969279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:31.969133 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:09:32.969636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:32.969603 4862 patch_prober.go:29] interesting pod/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:09:32.970125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:32.969699 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:09:33.969272 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:33.969220 4862 patch_prober.go:29] interesting pod/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:09:33.969449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:33.969297 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 18:09:35.046247 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:35.046218 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:35.046247 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:35.046638 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:35.048336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:35.048311 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:09:35.947992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:35.947969 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:43.318491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:43.318449 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:09:47.691669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:47.691631 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm"] Oct 30 18:09:49.868004 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:09:49.867970 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:09:50.166579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.166494 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm"] Oct 30 18:09:50.166748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.166720 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" podUID="099b5d37-50b2-4704-9113-208a7361c6f7" containerName="maas-api" containerID="cri-o://9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa" gracePeriod=30 Oct 30 18:09:50.429655 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.429590 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:50.546858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.546830 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-maas-api-tls\") pod \"099b5d37-50b2-4704-9113-208a7361c6f7\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " Oct 30 18:09:50.547028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.546871 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099b5d37-50b2-4704-9113-208a7361c6f7-kube-api-access-8zrjn\" (UniqueName: \"kubernetes.io/projected/099b5d37-50b2-4704-9113-208a7361c6f7-kube-api-access-8zrjn\") pod \"099b5d37-50b2-4704-9113-208a7361c6f7\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " Oct 30 18:09:50.547028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.546918 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099b5d37-50b2-4704-9113-208a7361c6f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/099b5d37-50b2-4704-9113-208a7361c6f7-openshift-service-ca\") pod \"099b5d37-50b2-4704-9113-208a7361c6f7\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " Oct 30 18:09:50.547028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.547004 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-metrics-certs\") pod \"099b5d37-50b2-4704-9113-208a7361c6f7\" (UID: \"099b5d37-50b2-4704-9113-208a7361c6f7\") " Oct 30 18:09:50.547336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.547314 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099b5d37-50b2-4704-9113-208a7361c6f7-openshift-service-ca" pod "099b5d37-50b2-4704-9113-208a7361c6f7" (UID: "099b5d37-50b2-4704-9113-208a7361c6f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:50.549134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.549105 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-maas-api-tls" pod "099b5d37-50b2-4704-9113-208a7361c6f7" (UID: "099b5d37-50b2-4704-9113-208a7361c6f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:50.549250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.549156 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-metrics-certs" pod "099b5d37-50b2-4704-9113-208a7361c6f7" (UID: "099b5d37-50b2-4704-9113-208a7361c6f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:50.549250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.549167 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099b5d37-50b2-4704-9113-208a7361c6f7-kube-api-access-8zrjn" pod "099b5d37-50b2-4704-9113-208a7361c6f7" (UID: "099b5d37-50b2-4704-9113-208a7361c6f7"). InnerVolumeSpecName "kube-api-access-8zrjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:50.648404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.648379 4862 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-metrics-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:50.648404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.648402 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/099b5d37-50b2-4704-9113-208a7361c6f7-maas-api-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:50.648404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.648411 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrjn\" (UniqueName: \"kubernetes.io/projected/099b5d37-50b2-4704-9113-208a7361c6f7-kube-api-access-8zrjn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:50.648605 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:50.648420 4862 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/099b5d37-50b2-4704-9113-208a7361c6f7-openshift-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:51.067110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:51.067074 4862 generic.go:356] "Generic (PLEG): container finished" podID="099b5d37-50b2-4704-9113-208a7361c6f7" containerID="9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa" exitCode=0 Oct 30 18:09:51.067526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:51.067116 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" Oct 30 18:09:51.067526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:51.067159 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" event={"ID":"099b5d37-50b2-4704-9113-208a7361c6f7","Type":"ContainerDied","Data":"9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa"} Oct 30 18:09:51.067526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:51.067211 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm" event={"ID":"099b5d37-50b2-4704-9113-208a7361c6f7","Type":"ContainerDied","Data":"30a6bbc5e183bf5adba405896ba4ec99a18cf309376f3a9d5b6c84a6497a6144"} Oct 30 18:09:51.067526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:51.067238 4862 scope.go:122] "RemoveContainer" containerID="9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa" Oct 30 18:09:51.076630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:51.076608 4862 scope.go:122] "RemoveContainer" containerID="9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa" Oct 30 18:09:51.076930 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:09:51.076901 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa\": container with ID starting with 9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa not found: ID does not exist" containerID="9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa" Oct 30 18:09:51.077020 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:51.076940 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa"} err="failed to get container status \"9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa\": rpc error: code = NotFound desc = could not find container \"9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa\": container with ID starting with 9cc123fccf3ee9569124ce82397431a7e9f89e224f2461250014a78bc25210fa not found: ID does not exist" Oct 30 18:09:51.116030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:51.116012 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm"] Oct 30 18:09:51.128225 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:51.128207 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a521cf49-59c877df55-nt8jm"] Oct 30 18:09:52.275669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.275566 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2"] Oct 30 18:09:52.276286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.275811 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" podUID="2f89a7c6-e6be-47a7-b5d1-35d08647bb30" containerName="maas-api" containerID="cri-o://277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274" gracePeriod=30 Oct 30 18:09:52.533468 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.533407 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:52.666611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.666583 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-kube-api-access-pqlps\" (UniqueName: \"kubernetes.io/projected/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-kube-api-access-pqlps\") pod \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " Oct 30 18:09:52.666768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.666640 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-maas-api-tls\") pod \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " Oct 30 18:09:52.666768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.666698 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-metrics-certs\") pod \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " Oct 30 18:09:52.666899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.666826 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-openshift-service-ca\") pod \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\" (UID: \"2f89a7c6-e6be-47a7-b5d1-35d08647bb30\") " Oct 30 18:09:52.667301 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.667215 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-openshift-service-ca" pod "2f89a7c6-e6be-47a7-b5d1-35d08647bb30" (UID: "2f89a7c6-e6be-47a7-b5d1-35d08647bb30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:52.668866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.668835 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-kube-api-access-pqlps" pod "2f89a7c6-e6be-47a7-b5d1-35d08647bb30" (UID: "2f89a7c6-e6be-47a7-b5d1-35d08647bb30"). InnerVolumeSpecName "kube-api-access-pqlps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:52.669002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.668864 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-metrics-certs" pod "2f89a7c6-e6be-47a7-b5d1-35d08647bb30" (UID: "2f89a7c6-e6be-47a7-b5d1-35d08647bb30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:52.669002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.668885 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-maas-api-tls" pod "2f89a7c6-e6be-47a7-b5d1-35d08647bb30" (UID: "2f89a7c6-e6be-47a7-b5d1-35d08647bb30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:52.691968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.691934 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" podUID="af4c61c9-855f-4083-aa0b-455fd6cdc719" containerName="payload-processing" containerID="cri-o://3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b" gracePeriod=25 Oct 30 18:09:52.767855 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.767826 4862 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-openshift-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:52.767855 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.767852 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqlps\" (UniqueName: \"kubernetes.io/projected/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-kube-api-access-pqlps\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:52.767855 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.767862 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-maas-api-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:52.768072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.767870 4862 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f89a7c6-e6be-47a7-b5d1-35d08647bb30-metrics-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:52.937616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.937592 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:52.987387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.987356 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m"] Oct 30 18:09:52.987913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.987896 4862 cpu_manager.go:414] "containerMap: removing container" podUID="099b5d37-50b2-4704-9113-208a7361c6f7" containerName="maas-api" Oct 30 18:09:52.988002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.987916 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099b5d37-50b2-4704-9113-208a7361c6f7" containerName="maas-api" Oct 30 18:09:52.988002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.987931 4862 cpu_manager.go:414] "containerMap: removing container" podUID="af4c61c9-855f-4083-aa0b-455fd6cdc719" containerName="payload-processing" Oct 30 18:09:52.988002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.987940 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4c61c9-855f-4083-aa0b-455fd6cdc719" containerName="payload-processing" Oct 30 18:09:52.988002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.987952 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2f89a7c6-e6be-47a7-b5d1-35d08647bb30" containerName="maas-api" Oct 30 18:09:52.988002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.987959 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f89a7c6-e6be-47a7-b5d1-35d08647bb30" containerName="maas-api" Oct 30 18:09:52.988249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.988071 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="099b5d37-50b2-4704-9113-208a7361c6f7" containerName="maas-api" Oct 30 18:09:52.988249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.988083 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4c61c9-855f-4083-aa0b-455fd6cdc719" containerName="payload-processing" Oct 30 18:09:52.988249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.988092 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f89a7c6-e6be-47a7-b5d1-35d08647bb30" containerName="maas-api" Oct 30 18:09:52.990188 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:52.990107 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.007882 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.007860 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-921450-openshift-default-dockercfg-rt5dw\"" Oct 30 18:09:53.007882 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.007864 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m"] Oct 30 18:09:53.061680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.061600 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099b5d37-50b2-4704-9113-208a7361c6f7" path="/var/lib/kubelet/pods/099b5d37-50b2-4704-9113-208a7361c6f7/volumes" Oct 30 18:09:53.070805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.070783 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4c61c9-855f-4083-aa0b-455fd6cdc719-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4c61c9-855f-4083-aa0b-455fd6cdc719-plugins-config-volume\") pod \"af4c61c9-855f-4083-aa0b-455fd6cdc719\" (UID: \"af4c61c9-855f-4083-aa0b-455fd6cdc719\") " Oct 30 18:09:53.070960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.070859 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4c61c9-855f-4083-aa0b-455fd6cdc719-kube-api-access-mxcmq\" (UniqueName: \"kubernetes.io/projected/af4c61c9-855f-4083-aa0b-455fd6cdc719-kube-api-access-mxcmq\") pod \"af4c61c9-855f-4083-aa0b-455fd6cdc719\" (UID: \"af4c61c9-855f-4083-aa0b-455fd6cdc719\") " Oct 30 18:09:53.071138 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071100 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.071200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071136 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtg96\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-kube-api-access-wtg96\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.071200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071167 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.071200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071174 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4c61c9-855f-4083-aa0b-455fd6cdc719-plugins-config-volume" pod "af4c61c9-855f-4083-aa0b-455fd6cdc719" (UID: "af4c61c9-855f-4083-aa0b-455fd6cdc719"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:53.071200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071195 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.071452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071312 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.071452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071417 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.071557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071493 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.071557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071524 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.071724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071634 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.071724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.071699 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4c61c9-855f-4083-aa0b-455fd6cdc719-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:53.073741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.073720 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4c61c9-855f-4083-aa0b-455fd6cdc719-kube-api-access-mxcmq" pod "af4c61c9-855f-4083-aa0b-455fd6cdc719" (UID: "af4c61c9-855f-4083-aa0b-455fd6cdc719"). InnerVolumeSpecName "kube-api-access-mxcmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:53.079251 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.079222 4862 generic.go:356] "Generic (PLEG): container finished" podID="af4c61c9-855f-4083-aa0b-455fd6cdc719" containerID="3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b" exitCode=0 Oct 30 18:09:53.079363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.079303 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" Oct 30 18:09:53.079363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.079314 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" event={"ID":"af4c61c9-855f-4083-aa0b-455fd6cdc719","Type":"ContainerDied","Data":"3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b"} Oct 30 18:09:53.079475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.079373 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm" event={"ID":"af4c61c9-855f-4083-aa0b-455fd6cdc719","Type":"ContainerDied","Data":"72518f616aea444fb99fb713a1f77d3d5407f2b32aef2f1f01aa8110e517c2d7"} Oct 30 18:09:53.079475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.079404 4862 scope.go:122] "RemoveContainer" containerID="3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b" Oct 30 18:09:53.089742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.089635 4862 generic.go:356] "Generic (PLEG): container finished" podID="2f89a7c6-e6be-47a7-b5d1-35d08647bb30" containerID="277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274" exitCode=0 Oct 30 18:09:53.089742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.089688 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" event={"ID":"2f89a7c6-e6be-47a7-b5d1-35d08647bb30","Type":"ContainerDied","Data":"277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274"} Oct 30 18:09:53.090062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.089769 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" event={"ID":"2f89a7c6-e6be-47a7-b5d1-35d08647bb30","Type":"ContainerDied","Data":"47a449b0ed19bd4b2f5f54888372a56c73bb5b8493949770c42bf3b2dd3c1900"} Oct 30 18:09:53.090062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.089934 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2" Oct 30 18:09:53.098692 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.098661 4862 scope.go:122] "RemoveContainer" containerID="3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b" Oct 30 18:09:53.098953 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:09:53.098931 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b\": container with ID starting with 3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b not found: ID does not exist" containerID="3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b" Oct 30 18:09:53.099038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.098968 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b"} err="failed to get container status \"3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b\": rpc error: code = NotFound desc = could not find container \"3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b\": container with ID starting with 3df681bb471d900ccc399643339bd4846031440f30290845afa77513de80561b not found: ID does not exist" Oct 30 18:09:53.099038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.098984 4862 scope.go:122] "RemoveContainer" containerID="277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274" Oct 30 18:09:53.108435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.108409 4862 scope.go:122] "RemoveContainer" containerID="277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274" Oct 30 18:09:53.108710 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:09:53.108694 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274\": container with ID starting with 277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274 not found: ID does not exist" containerID="277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274" Oct 30 18:09:53.108767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.108717 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274"} err="failed to get container status \"277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274\": rpc error: code = NotFound desc = could not find container \"277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274\": container with ID starting with 277b809f8b741da204e34f403a7fd1db59da5d785303ea1ea03e6f3596f17274 not found: ID does not exist" Oct 30 18:09:53.132361 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.132335 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2"] Oct 30 18:09:53.140003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.139984 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-594f46d3-5889c999df-78zz2"] Oct 30 18:09:53.172460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172441 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.172556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172471 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtg96\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-kube-api-access-wtg96\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.172556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172491 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.172556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172512 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.172556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172542 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.172796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172568 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.172796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172604 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.172796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172649 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.172796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172710 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.172796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.172766 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxcmq\" (UniqueName: \"kubernetes.io/projected/af4c61c9-855f-4083-aa0b-455fd6cdc719-kube-api-access-mxcmq\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:09:53.173506 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.173457 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.173585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.173519 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.174071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.174008 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.174227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.174146 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.174227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.174193 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.175205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.175184 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.175808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.175791 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.204787 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.204764 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm"] Oct 30 18:09:53.205208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.205186 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.205443 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.205428 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtg96\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-kube-api-access-wtg96\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.206253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.206231 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a521cf49-6ccf69fbd-bvmjm"] Oct 30 18:09:53.304680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.304651 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:53.630012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.629990 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m"] Oct 30 18:09:53.632432 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:09:53.632403 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1735e3_e2ea_4e64_a372_e88738de6b4a.slice/crio-698feb0f1119344cdad2196e5d0704364ba218fa886d483a0f9c1296f2df5f26 WatchSource:0}: Error finding container 698feb0f1119344cdad2196e5d0704364ba218fa886d483a0f9c1296f2df5f26: Status 404 returned error can't find the container with id 698feb0f1119344cdad2196e5d0704364ba218fa886d483a0f9c1296f2df5f26 Oct 30 18:09:53.634744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.634714 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:53.634842 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.634792 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:53.634842 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:53.634834 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:54.096490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.096449 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:09:54.098402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.098369 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" event={"ID":"1c1735e3-e2ea-4e64-a372-e88738de6b4a","Type":"ContainerStarted","Data":"d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25"} Oct 30 18:09:54.098537 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.098416 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" event={"ID":"1c1735e3-e2ea-4e64-a372-e88738de6b4a","Type":"ContainerStarted","Data":"698feb0f1119344cdad2196e5d0704364ba218fa886d483a0f9c1296f2df5f26"} Oct 30 18:09:54.154484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.154434 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podStartSLOduration=2.154421312 podStartE2EDuration="2.154421312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:54.152344439 +0000 UTC m=+693.201389352" watchObservedRunningTime="2026-10-30 18:09:54.154421312 +0000 UTC m=+693.203466225" Oct 30 18:09:54.304984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.304955 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:54.310038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.310016 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:54.506737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.506475 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2"] Oct 30 18:09:54.510022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.509363 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.534363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.534335 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2"] Oct 30 18:09:54.536729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.536710 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-45dc34-openshift-default-dockercfg-w86md\"" Oct 30 18:09:54.599279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.599235 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.599429 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.599301 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca9be35-a071-480e-97bb-68eedec2f8c6-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.599477 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.599414 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.599514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.599492 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpvn\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-kube-api-access-dkpvn\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.599588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.599572 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.599649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.599631 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.599687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.599669 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.599764 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.599753 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.599808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.599788 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701195 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701251 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701306 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701331 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca9be35-a071-480e-97bb-68eedec2f8c6-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701353 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701382 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpvn\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-kube-api-access-dkpvn\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701435 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701472 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701508 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701863 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701819 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701881 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.701996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.701974 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.702135 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.702085 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.702306 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.702191 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca9be35-a071-480e-97bb-68eedec2f8c6-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.703984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.703960 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.704166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.704147 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.721917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.721896 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpvn\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-kube-api-access-dkpvn\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.722097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.722076 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:54.823987 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:54.823959 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:55.062200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:55.062164 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f89a7c6-e6be-47a7-b5d1-35d08647bb30" path="/var/lib/kubelet/pods/2f89a7c6-e6be-47a7-b5d1-35d08647bb30/volumes" Oct 30 18:09:55.062679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:55.062661 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4c61c9-855f-4083-aa0b-455fd6cdc719" path="/var/lib/kubelet/pods/af4c61c9-855f-4083-aa0b-455fd6cdc719/volumes" Oct 30 18:09:55.103453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:55.103397 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:55.104391 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:55.104373 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:09:55.138305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:55.138288 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2"] Oct 30 18:09:55.140498 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:09:55.140471 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca9be35_a071_480e_97bb_68eedec2f8c6.slice/crio-08b4a6bea3c9fd75e4e8de57d0b5d98fae0724036b422c15d364c77ea4a33f77 WatchSource:0}: Error finding container 08b4a6bea3c9fd75e4e8de57d0b5d98fae0724036b422c15d364c77ea4a33f77: Status 404 returned error can't find the container with id 08b4a6bea3c9fd75e4e8de57d0b5d98fae0724036b422c15d364c77ea4a33f77 Oct 30 18:09:55.142912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:55.142877 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:55.142974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:55.142960 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:55.143016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:55.143005 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:09:56.110290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:56.110237 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" event={"ID":"6ca9be35-a071-480e-97bb-68eedec2f8c6","Type":"ContainerStarted","Data":"7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde"} Oct 30 18:09:56.112595 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:56.110315 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" event={"ID":"6ca9be35-a071-480e-97bb-68eedec2f8c6","Type":"ContainerStarted","Data":"08b4a6bea3c9fd75e4e8de57d0b5d98fae0724036b422c15d364c77ea4a33f77"} Oct 30 18:09:56.184459 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:56.184397 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" podStartSLOduration=2.184379051 podStartE2EDuration="2.184379051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:56.170666604 +0000 UTC m=+695.219711517" watchObservedRunningTime="2026-10-30 18:09:56.184379051 +0000 UTC m=+695.233423964" Oct 30 18:09:56.825167 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:56.825136 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:56.829946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:56.829924 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:57.114515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:57.114419 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:57.115366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:57.115352 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:09:59.554987 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:59.554952 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h"] Oct 30 18:09:59.555453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:09:59.555207 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" containerID="cri-o://053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f" gracePeriod=30 Oct 30 18:10:02.911542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:02.911501 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:10:04.844353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.844330 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:10:04.901528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901498 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-certs\") pod \"cd568017-d361-47cc-b9c3-9a0425d72ba8\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " Oct 30 18:10:04.901701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901580 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-data\") pod \"cd568017-d361-47cc-b9c3-9a0425d72ba8\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " Oct 30 18:10:04.901701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901606 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-podinfo\") pod \"cd568017-d361-47cc-b9c3-9a0425d72ba8\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " Oct 30 18:10:04.901701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901657 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-token\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-token\") pod \"cd568017-d361-47cc-b9c3-9a0425d72ba8\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " Oct 30 18:10:04.901876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901740 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-certs" pod "cd568017-d361-47cc-b9c3-9a0425d72ba8" (UID: "cd568017-d361-47cc-b9c3-9a0425d72ba8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:04.901876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901782 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd568017-d361-47cc-b9c3-9a0425d72ba8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd568017-d361-47cc-b9c3-9a0425d72ba8-istiod-ca-cert\") pod \"cd568017-d361-47cc-b9c3-9a0425d72ba8\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " Oct 30 18:10:04.901876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901833 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-data" pod "cd568017-d361-47cc-b9c3-9a0425d72ba8" (UID: "cd568017-d361-47cc-b9c3-9a0425d72ba8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:04.901876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901834 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-kube-api-access-fdkmn\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-kube-api-access-fdkmn\") pod \"cd568017-d361-47cc-b9c3-9a0425d72ba8\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " Oct 30 18:10:04.902080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901884 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-credential-socket\") pod \"cd568017-d361-47cc-b9c3-9a0425d72ba8\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " Oct 30 18:10:04.902080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.901912 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-envoy\") pod \"cd568017-d361-47cc-b9c3-9a0425d72ba8\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " Oct 30 18:10:04.902080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.902003 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-socket\") pod \"cd568017-d361-47cc-b9c3-9a0425d72ba8\" (UID: \"cd568017-d361-47cc-b9c3-9a0425d72ba8\") " Oct 30 18:10:04.902237 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.902122 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-credential-socket" pod "cd568017-d361-47cc-b9c3-9a0425d72ba8" (UID: "cd568017-d361-47cc-b9c3-9a0425d72ba8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:04.902237 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.902182 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd568017-d361-47cc-b9c3-9a0425d72ba8-istiod-ca-cert" pod "cd568017-d361-47cc-b9c3-9a0425d72ba8" (UID: "cd568017-d361-47cc-b9c3-9a0425d72ba8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:10:04.902404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.902344 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-socket" pod "cd568017-d361-47cc-b9c3-9a0425d72ba8" (UID: "cd568017-d361-47cc-b9c3-9a0425d72ba8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:04.902678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.902656 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:10:04.902739 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.902684 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd568017-d361-47cc-b9c3-9a0425d72ba8-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:10:04.902739 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.902700 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:10:04.902739 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.902716 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:10:04.902739 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.902730 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:10:04.904452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.904426 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-podinfo" pod "cd568017-d361-47cc-b9c3-9a0425d72ba8" (UID: "cd568017-d361-47cc-b9c3-9a0425d72ba8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:10:04.904590 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.904569 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-kube-api-access-fdkmn" pod "cd568017-d361-47cc-b9c3-9a0425d72ba8" (UID: "cd568017-d361-47cc-b9c3-9a0425d72ba8"). InnerVolumeSpecName "kube-api-access-fdkmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:04.904638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.904581 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-token" pod "cd568017-d361-47cc-b9c3-9a0425d72ba8" (UID: "cd568017-d361-47cc-b9c3-9a0425d72ba8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:04.905451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:04.905423 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-envoy" pod "cd568017-d361-47cc-b9c3-9a0425d72ba8" (UID: "cd568017-d361-47cc-b9c3-9a0425d72ba8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:05.003934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.003906 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:10:05.003934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.003932 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:10:05.003934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.003942 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdkmn\" (UniqueName: \"kubernetes.io/projected/cd568017-d361-47cc-b9c3-9a0425d72ba8-kube-api-access-fdkmn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:10:05.004175 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.003952 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd568017-d361-47cc-b9c3-9a0425d72ba8-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:10:05.160139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.160063 4862 generic.go:356] "Generic (PLEG): container finished" podID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerID="053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f" exitCode=0 Oct 30 18:10:05.160139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.160110 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" Oct 30 18:10:05.160377 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.160138 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" event={"ID":"cd568017-d361-47cc-b9c3-9a0425d72ba8","Type":"ContainerDied","Data":"053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f"} Oct 30 18:10:05.160377 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.160202 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h" event={"ID":"cd568017-d361-47cc-b9c3-9a0425d72ba8","Type":"ContainerDied","Data":"094e08e2b86df03c2a95784af0f7923cf97c5d3e337840e5f3704a444781994b"} Oct 30 18:10:05.160377 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.160232 4862 scope.go:122] "RemoveContainer" containerID="053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f" Oct 30 18:10:05.168833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.168818 4862 scope.go:122] "RemoveContainer" containerID="046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad" Oct 30 18:10:05.176507 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.176491 4862 scope.go:122] "RemoveContainer" containerID="053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f" Oct 30 18:10:05.176761 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:10:05.176744 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f\": container with ID starting with 053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f not found: ID does not exist" containerID="053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f" Oct 30 18:10:05.176808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.176769 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f"} err="failed to get container status \"053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f\": rpc error: code = NotFound desc = could not find container \"053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f\": container with ID starting with 053b4e6f50789ca90300bb6eeee9b4a4ac357d4f6e8bc4c46366aca1e6c73c9f not found: ID does not exist" Oct 30 18:10:05.176808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.176783 4862 scope.go:122] "RemoveContainer" containerID="046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad" Oct 30 18:10:05.177027 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:10:05.176993 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad\": container with ID starting with 046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad not found: ID does not exist" containerID="046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad" Oct 30 18:10:05.177069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.177030 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad"} err="failed to get container status \"046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad\": rpc error: code = NotFound desc = could not find container \"046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad\": container with ID starting with 046bb0c6fb86935c6e4715f91de7feee23ce8abd08a17cf670c63151eddc40ad not found: ID does not exist" Oct 30 18:10:05.212768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.212747 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h"] Oct 30 18:10:05.224962 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:05.224940 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-57bf5177-gw-openshift-default-dfdd4fc-5cl7h"] Oct 30 18:10:07.061039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:07.061010 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" path="/var/lib/kubelet/pods/cd568017-d361-47cc-b9c3-9a0425d72ba8/volumes" Oct 30 18:10:12.986768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:12.986734 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:10:17.158237 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.158196 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l"] Oct 30 18:10:17.158785 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.158759 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" Oct 30 18:10:17.158872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.158793 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" Oct 30 18:10:17.158971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.158957 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" Oct 30 18:10:17.159046 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.158975 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" Oct 30 18:10:17.164206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.164185 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.192152 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.192130 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-380c08-serving-cert\"" Oct 30 18:10:17.195576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.195551 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l"] Oct 30 18:10:17.259518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.259483 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4gc\" (UniqueName: \"kubernetes.io/projected/83501714-680a-4c27-bfef-fd5e65bc12c9-kube-api-access-5b4gc\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.259703 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.259535 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83501714-680a-4c27-bfef-fd5e65bc12c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.259703 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.259657 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.259834 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.259726 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-metrics-certs\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.360630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.360594 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4gc\" (UniqueName: \"kubernetes.io/projected/83501714-680a-4c27-bfef-fd5e65bc12c9-kube-api-access-5b4gc\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.360630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.360632 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83501714-680a-4c27-bfef-fd5e65bc12c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.360894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.360668 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.360894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.360781 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-metrics-certs\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.361786 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.361748 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83501714-680a-4c27-bfef-fd5e65bc12c9-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.363948 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.363924 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-metrics-certs\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.364134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.364111 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-maas-api-tls\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.388943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.388913 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4gc\" (UniqueName: \"kubernetes.io/projected/83501714-680a-4c27-bfef-fd5e65bc12c9-kube-api-access-5b4gc\") pod \"maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.486528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.486447 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:17.489555 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.489513 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz"] Oct 30 18:10:17.490323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.490299 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" Oct 30 18:10:17.490323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.490324 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd568017-d361-47cc-b9c3-9a0425d72ba8" containerName="istio-proxy" Oct 30 18:10:17.494533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.494515 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:17.509717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.509678 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-fe1374-dockercfg-wvc5w\"" Oct 30 18:10:17.510203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.510185 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz"] Oct 30 18:10:17.511565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.511542 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-fe1374\"" Oct 30 18:10:17.567426 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.567385 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92wx6\" (UniqueName: \"kubernetes.io/projected/8eae894d-325c-48b1-bfd3-d016d9d2a741-kube-api-access-92wx6\") pod \"payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz\" (UID: \"8eae894d-325c-48b1-bfd3-d016d9d2a741\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:17.567594 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.567526 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eae894d-325c-48b1-bfd3-d016d9d2a741-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz\" (UID: \"8eae894d-325c-48b1-bfd3-d016d9d2a741\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:17.669076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.669032 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eae894d-325c-48b1-bfd3-d016d9d2a741-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz\" (UID: \"8eae894d-325c-48b1-bfd3-d016d9d2a741\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:17.669279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.669112 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92wx6\" (UniqueName: \"kubernetes.io/projected/8eae894d-325c-48b1-bfd3-d016d9d2a741-kube-api-access-92wx6\") pod \"payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz\" (UID: \"8eae894d-325c-48b1-bfd3-d016d9d2a741\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:17.670011 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.669987 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eae894d-325c-48b1-bfd3-d016d9d2a741-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz\" (UID: \"8eae894d-325c-48b1-bfd3-d016d9d2a741\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:17.688859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.688836 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92wx6\" (UniqueName: \"kubernetes.io/projected/8eae894d-325c-48b1-bfd3-d016d9d2a741-kube-api-access-92wx6\") pod \"payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz\" (UID: \"8eae894d-325c-48b1-bfd3-d016d9d2a741\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:17.855465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.849154 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l"] Oct 30 18:10:17.857762 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:10:17.857728 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83501714_680a_4c27_bfef_fd5e65bc12c9.slice/crio-31df38f208df065536d58a3f7b03ccb9810dcb54f7baf33e4408d2ea9e2ae800 WatchSource:0}: Error finding container 31df38f208df065536d58a3f7b03ccb9810dcb54f7baf33e4408d2ea9e2ae800: Status 404 returned error can't find the container with id 31df38f208df065536d58a3f7b03ccb9810dcb54f7baf33e4408d2ea9e2ae800 Oct 30 18:10:17.870671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:17.870645 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:18.169214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:18.169167 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz"] Oct 30 18:10:18.171752 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:10:18.171660 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eae894d_325c_48b1_bfd3_d016d9d2a741.slice/crio-91a1885ab9fe194b9f9c8ed6d9adf26408913a7c8d126737de1835fd122eb966 WatchSource:0}: Error finding container 91a1885ab9fe194b9f9c8ed6d9adf26408913a7c8d126737de1835fd122eb966: Status 404 returned error can't find the container with id 91a1885ab9fe194b9f9c8ed6d9adf26408913a7c8d126737de1835fd122eb966 Oct 30 18:10:18.253809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:18.253472 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" event={"ID":"8eae894d-325c-48b1-bfd3-d016d9d2a741","Type":"ContainerStarted","Data":"91a1885ab9fe194b9f9c8ed6d9adf26408913a7c8d126737de1835fd122eb966"} Oct 30 18:10:18.257862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:18.257733 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" event={"ID":"83501714-680a-4c27-bfef-fd5e65bc12c9","Type":"ContainerStarted","Data":"31df38f208df065536d58a3f7b03ccb9810dcb54f7baf33e4408d2ea9e2ae800"} Oct 30 18:10:19.264865 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.264831 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" event={"ID":"83501714-680a-4c27-bfef-fd5e65bc12c9","Type":"ContainerStarted","Data":"0444f4eff4eb32ac10fd2717a48e92ec35b35fe22c86eaa0bedbce3c90cb8331"} Oct 30 18:10:19.265610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.264901 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:19.266557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.266534 4862 generic.go:356] "Generic (PLEG): container finished" podID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerID="d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25" exitCode=0 Oct 30 18:10:19.266646 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.266602 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" event={"ID":"1c1735e3-e2ea-4e64-a372-e88738de6b4a","Type":"ContainerDied","Data":"d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25"} Oct 30 18:10:19.266949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.266935 4862 scope.go:122] "RemoveContainer" containerID="d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25" Oct 30 18:10:19.268329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.268300 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:19.268481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.268381 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:19.268481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.268395 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" event={"ID":"8eae894d-325c-48b1-bfd3-d016d9d2a741","Type":"ContainerStarted","Data":"ee604b44bbac10caa081bac25f24ecff1b876aa6b59056a9263ce011af3217e8"} Oct 30 18:10:19.268481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.268416 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:19.268481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.268446 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:19.314327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.313505 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" podStartSLOduration=1.890361141 podStartE2EDuration="2.313491131s" totalImagesPullingTime="423.12999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:19.311702468 +0000 UTC m=+718.360747381" watchObservedRunningTime="2026-10-30 18:10:19.313491131 +0000 UTC m=+718.362536048" Oct 30 18:10:19.462640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.462581 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" podStartSLOduration=2.462563378 podStartE2EDuration="2.462563378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:19.459765476 +0000 UTC m=+718.508810390" watchObservedRunningTime="2026-10-30 18:10:19.462563378 +0000 UTC m=+718.511608293" Oct 30 18:10:19.791643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:19.791607 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:10:19.905457 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:10:19.905418 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:10:20.276824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:20.276792 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" event={"ID":"1c1735e3-e2ea-4e64-a372-e88738de6b4a","Type":"ContainerStarted","Data":"bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231"} Oct 30 18:10:20.304935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:20.304912 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:10:20.305575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:20.305553 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 18:10:20.305675 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:20.305594 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 18:10:21.305807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:21.305771 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 18:10:21.306207 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:21.305841 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 18:10:22.305370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.305334 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 18:10:22.305544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.305396 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 18:10:22.355287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.355046 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt"] Oct 30 18:10:22.362076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.362049 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.378569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.378547 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt"] Oct 30 18:10:22.379139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.379066 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-fe1374-serving-cert\"" Oct 30 18:10:22.421318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.421290 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.421476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.421341 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd6xl\" (UniqueName: \"kubernetes.io/projected/6a1b5627-547f-48aa-b59f-36bcdba7172e-kube-api-access-nd6xl\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.421476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.421441 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b5627-547f-48aa-b59f-36bcdba7172e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.421603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.421535 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-metrics-certs\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.466947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.466915 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl"] Oct 30 18:10:22.470575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.470552 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.486067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.485838 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"openshift-service-ca.crt\"" Oct 30 18:10:22.486067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.485948 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"default-dockercfg-phb79\"" Oct 30 18:10:22.486067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.485998 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"simulator-w2-45dc34-kserve-self-signed-certs\"" Oct 30 18:10:22.486370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.486126 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"kube-root-ca.crt\"" Oct 30 18:10:22.496291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.496232 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl"] Oct 30 18:10:22.522382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522356 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd6xl\" (UniqueName: \"kubernetes.io/projected/6a1b5627-547f-48aa-b59f-36bcdba7172e-kube-api-access-nd6xl\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.522502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522392 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b5627-547f-48aa-b59f-36bcdba7172e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.522502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522432 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvqfb\" (UniqueName: \"kubernetes.io/projected/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-kube-api-access-pvqfb\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.522502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522457 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-home\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.522502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522479 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-metrics-certs\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.522502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522499 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-model-cache\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.522671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522582 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tmp-dir\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.522671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522633 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.522671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522660 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tls-certs\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.522779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.522676 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-dshm\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.522779 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:10:22.522731 4862 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-serving-cert: secret "maas-api-e2e-worker-w4-fe1374-serving-cert" not found Oct 30 18:10:22.522848 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:10:22.522791 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls podName:6a1b5627-547f-48aa-b59f-36bcdba7172e nodeName:}" failed. No retries permitted until 2026-10-30 18:10:23.022776649 +0000 UTC m=+722.071821540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls") pod "maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" (UID: "6a1b5627-547f-48aa-b59f-36bcdba7172e") : secret "maas-api-e2e-worker-w4-fe1374-serving-cert" not found Oct 30 18:10:22.523231 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.523207 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b5627-547f-48aa-b59f-36bcdba7172e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.525310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.525247 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-metrics-certs\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.563805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.563734 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd6xl\" (UniqueName: \"kubernetes.io/projected/6a1b5627-547f-48aa-b59f-36bcdba7172e-kube-api-access-nd6xl\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:22.623384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.623348 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvqfb\" (UniqueName: \"kubernetes.io/projected/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-kube-api-access-pvqfb\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.623584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.623408 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-home\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.623584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.623449 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-model-cache\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.623584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.623477 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tmp-dir\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.623584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.623547 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tls-certs\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.623584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.623574 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-dshm\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.623886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.623860 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-model-cache\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.626374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.626350 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-dshm\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.626538 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.626412 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tls-certs\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.632285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.631527 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tmp-dir\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.632522 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.632498 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-home\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.669118 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.668929 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvqfb\" (UniqueName: \"kubernetes.io/projected/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-kube-api-access-pvqfb\") pod \"simulator-w2-45dc34-kserve-759f94499d-mx4hl\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:22.785296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:22.785046 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:23.057295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.028578 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:23.057295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.033142 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:23.120982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.120948 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl"] Oct 30 18:10:23.279430 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.279398 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:23.296678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.296639 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" event={"ID":"09fe261a-c563-4a7d-a2c1-6e0c17242fe8","Type":"ContainerStarted","Data":"dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32"} Oct 30 18:10:23.296805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.296698 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" event={"ID":"09fe261a-c563-4a7d-a2c1-6e0c17242fe8","Type":"ContainerStarted","Data":"ebf5fb18c1553e0680567575958894a43e190961b0b05f35c02c11bedbd34a27"} Oct 30 18:10:23.305013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.304981 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 18:10:23.305121 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.305037 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 18:10:23.346563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.344637 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podStartSLOduration=1.344617884 podStartE2EDuration="1.344617884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:23.339513616 +0000 UTC m=+722.388558545" watchObservedRunningTime="2026-10-30 18:10:23.344617884 +0000 UTC m=+722.393662799" Oct 30 18:10:23.659861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.659777 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt"] Oct 30 18:10:23.786254 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:23.786215 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:24.306386 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:24.306338 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 18:10:24.306583 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:24.306411 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 18:10:24.307067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:24.307041 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" event={"ID":"6a1b5627-547f-48aa-b59f-36bcdba7172e","Type":"ContainerStarted","Data":"882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed"} Oct 30 18:10:24.307165 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:24.307084 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" event={"ID":"6a1b5627-547f-48aa-b59f-36bcdba7172e","Type":"ContainerStarted","Data":"de206d97676d98aee73fd438e7de197c0ee8ee23ba9615c6b301df6368eb7417"} Oct 30 18:10:24.307165 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:24.307120 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:25.107094 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.106893 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:10:25.282900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.282873 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 18:10:25.305807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.305777 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 18:10:25.306105 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.305852 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 18:10:25.335906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.335875 4862 generic.go:356] "Generic (PLEG): container finished" podID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerID="7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde" exitCode=0 Oct 30 18:10:25.336325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.336299 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" event={"ID":"6ca9be35-a071-480e-97bb-68eedec2f8c6","Type":"ContainerDied","Data":"7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde"} Oct 30 18:10:25.337309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.337238 4862 scope.go:122] "RemoveContainer" containerID="7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde" Oct 30 18:10:25.338657 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.338627 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:25.338748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.338700 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:25.338748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.338736 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:25.353042 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:25.353003 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" podStartSLOduration=3.027650931 podStartE2EDuration="3.352990333s" totalImagesPullingTime="325.339402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:24.34973072 +0000 UTC m=+723.398775635" watchObservedRunningTime="2026-10-30 18:10:25.352990333 +0000 UTC m=+724.402035245" Oct 30 18:10:26.309311 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:26.309255 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:10:26.310346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:26.310326 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:10:26.357834 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:26.357800 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" event={"ID":"6ca9be35-a071-480e-97bb-68eedec2f8c6","Type":"ContainerStarted","Data":"7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3"} Oct 30 18:10:26.824342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:26.824315 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:10:26.825426 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:26.825398 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:10:26.825554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:26.825450 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:10:27.114520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:27.114458 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:10:27.527080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:27.526958 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:10:27.824531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:27.824495 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:10:27.824677 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:27.824569 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:10:28.824331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:28.824293 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:10:28.824822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:28.824356 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:10:29.824978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:29.824949 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:10:29.825393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:29.825001 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:10:30.280386 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:30.280310 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 18:10:30.360836 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:30.360809 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 18:10:30.824672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:30.824641 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:10:30.824917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:30.824690 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:10:31.902422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:31.902389 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:10:31.903827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:31.903803 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:10:32.785930 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:32.785900 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:32.798231 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:32.798210 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:33.407342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:33.407315 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:10:37.165652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.165615 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65"] Oct 30 18:10:37.169683 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.169660 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.199559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.199226 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"default-dockercfg-p66x8\"" Oct 30 18:10:37.199559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.199345 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"openshift-service-ca.crt\"" Oct 30 18:10:37.199559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.199376 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"kube-root-ca.crt\"" Oct 30 18:10:37.199559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.199515 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"simulator-w3-de7e3a-kserve-self-signed-certs\"" Oct 30 18:10:37.201398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.201378 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65"] Oct 30 18:10:37.294698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.294663 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-tmp-dir\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.294852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.294707 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-home\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.294948 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.294845 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-model-cache\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.294989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.294957 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-dshm\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.295079 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.295064 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78zr\" (UniqueName: \"kubernetes.io/projected/ae7cb54c-237a-47d1-8b57-151748a41013-kube-api-access-m78zr\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.295123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.295090 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7cb54c-237a-47d1-8b57-151748a41013-tls-certs\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.396563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.396525 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78zr\" (UniqueName: \"kubernetes.io/projected/ae7cb54c-237a-47d1-8b57-151748a41013-kube-api-access-m78zr\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.396745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.396577 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7cb54c-237a-47d1-8b57-151748a41013-tls-certs\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.396745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.396640 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-tmp-dir\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.396745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.396677 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-home\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.396745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.396728 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-model-cache\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.396966 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.396791 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-dshm\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.397381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.397352 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-tmp-dir\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.398397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.397462 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-home\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.398845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.398767 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-model-cache\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.412388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.412361 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-dshm\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.420641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.420548 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78zr\" (UniqueName: \"kubernetes.io/projected/ae7cb54c-237a-47d1-8b57-151748a41013-kube-api-access-m78zr\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.422818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.422790 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7cb54c-237a-47d1-8b57-151748a41013-tls-certs\") pod \"simulator-w3-de7e3a-kserve-686c95ff4c-h9b65\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.485198 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.485169 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:37.781130 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:37.781111 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65"] Oct 30 18:10:37.783150 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:10:37.783126 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae7cb54c_237a_47d1_8b57_151748a41013.slice/crio-c638c1cc07d81b3b6e507456cdc80204457e44a5a94a1c4498c62fb5393e314d WatchSource:0}: Error finding container c638c1cc07d81b3b6e507456cdc80204457e44a5a94a1c4498c62fb5393e314d: Status 404 returned error can't find the container with id c638c1cc07d81b3b6e507456cdc80204457e44a5a94a1c4498c62fb5393e314d Oct 30 18:10:38.430217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:38.430182 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" event={"ID":"ae7cb54c-237a-47d1-8b57-151748a41013","Type":"ContainerStarted","Data":"bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5"} Oct 30 18:10:38.430634 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:38.430227 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" event={"ID":"ae7cb54c-237a-47d1-8b57-151748a41013","Type":"ContainerStarted","Data":"c638c1cc07d81b3b6e507456cdc80204457e44a5a94a1c4498c62fb5393e314d"} Oct 30 18:10:38.470994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:38.470950 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podStartSLOduration=1.47093917 podStartE2EDuration="1.47093917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:38.469891615 +0000 UTC m=+737.518936528" watchObservedRunningTime="2026-10-30 18:10:38.47093917 +0000 UTC m=+737.519984084" Oct 30 18:10:38.485652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:38.485625 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:39.279184 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.279154 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx"] Oct 30 18:10:39.282883 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.282862 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.301899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.301870 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx"] Oct 30 18:10:39.312244 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.312219 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"openshift-service-ca.crt\"" Oct 30 18:10:39.312244 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.312235 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"kube-root-ca.crt\"" Oct 30 18:10:39.312244 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.312228 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"default-dockercfg-nm4lf\"" Oct 30 18:10:39.312511 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.312306 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"simulator-w0-921450-kserve-self-signed-certs\"" Oct 30 18:10:39.416081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.416053 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlwzx\" (UniqueName: \"kubernetes.io/projected/3889cf6b-a606-47e0-a196-879e1b5b30f4-kube-api-access-dlwzx\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.416081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.416086 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-tmp-dir\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.416365 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.416118 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-home\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.416365 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.416327 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3889cf6b-a606-47e0-a196-879e1b5b30f4-tls-certs\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.416468 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.416413 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-dshm\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.416521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.416476 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-model-cache\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.517757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.517714 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-model-cache\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.518246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.517780 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlwzx\" (UniqueName: \"kubernetes.io/projected/3889cf6b-a606-47e0-a196-879e1b5b30f4-kube-api-access-dlwzx\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.518246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.517815 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-tmp-dir\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.518246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.517886 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-home\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.518246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.517955 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3889cf6b-a606-47e0-a196-879e1b5b30f4-tls-certs\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.518246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.518016 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-dshm\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.518246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.518191 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-model-cache\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.518604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.518516 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-tmp-dir\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.518663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.518629 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-home\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.520892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.520869 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-dshm\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.521609 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.521587 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3889cf6b-a606-47e0-a196-879e1b5b30f4-tls-certs\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.537840 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.537762 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlwzx\" (UniqueName: \"kubernetes.io/projected/3889cf6b-a606-47e0-a196-879e1b5b30f4-kube-api-access-dlwzx\") pod \"simulator-w0-921450-kserve-65969d5794-jnqkx\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.596114 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.596083 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:39.897273 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:39.897234 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx"] Oct 30 18:10:39.898905 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:10:39.898875 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3889cf6b_a606_47e0_a196_879e1b5b30f4.slice/crio-c83526fe102a550bde219c9089ac1308ff3d8f29d386752eb6ef348cd323d00c WatchSource:0}: Error finding container c83526fe102a550bde219c9089ac1308ff3d8f29d386752eb6ef348cd323d00c: Status 404 returned error can't find the container with id c83526fe102a550bde219c9089ac1308ff3d8f29d386752eb6ef348cd323d00c Oct 30 18:10:40.216594 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:40.216510 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:10:40.460288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:40.456278 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" event={"ID":"3889cf6b-a606-47e0-a196-879e1b5b30f4","Type":"ContainerStarted","Data":"70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8"} Oct 30 18:10:40.460288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:40.456323 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" event={"ID":"3889cf6b-a606-47e0-a196-879e1b5b30f4","Type":"ContainerStarted","Data":"c83526fe102a550bde219c9089ac1308ff3d8f29d386752eb6ef348cd323d00c"} Oct 30 18:10:40.597273 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:40.597222 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:46.823926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.823869 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podStartSLOduration=7.823853185 podStartE2EDuration="7.823853185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:40.520780093 +0000 UTC m=+739.569825006" watchObservedRunningTime="2026-10-30 18:10:46.823853185 +0000 UTC m=+745.872898097" Oct 30 18:10:46.824596 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.824042 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq"] Oct 30 18:10:46.827981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.827958 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.842746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.842721 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"kube-root-ca.crt\"" Oct 30 18:10:46.844746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.844709 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq"] Oct 30 18:10:46.855409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.855379 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"default-dockercfg-w2r27\"" Oct 30 18:10:46.855528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.855467 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"simulator-w1-380c08-kserve-self-signed-certs\"" Oct 30 18:10:46.895092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.895050 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cg4b\" (UniqueName: \"kubernetes.io/projected/8673e2ff-31e6-4172-ab13-8e5ea74713cf-kube-api-access-5cg4b\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.895233 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.895112 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-dshm\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.895333 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.895315 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-home\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.895376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.895345 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tmp-dir\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.895434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.895379 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tls-certs\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.895535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.895520 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-model-cache\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.942843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.942813 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"openshift-service-ca.crt\"" Oct 30 18:10:46.996318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.996284 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-home\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.996505 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.996325 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tmp-dir\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.996505 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.996356 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tls-certs\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.996505 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.996388 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-model-cache\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.996505 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.996431 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cg4b\" (UniqueName: \"kubernetes.io/projected/8673e2ff-31e6-4172-ab13-8e5ea74713cf-kube-api-access-5cg4b\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.996505 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.996467 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-dshm\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.996818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.996789 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-home\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.997020 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.996885 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-model-cache\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.997020 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.997013 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tmp-dir\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.998983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.998960 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-dshm\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:46.999212 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:46.999195 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tls-certs\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:47.015537 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:47.015511 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cg4b\" (UniqueName: \"kubernetes.io/projected/8673e2ff-31e6-4172-ab13-8e5ea74713cf-kube-api-access-5cg4b\") pod \"simulator-w1-380c08-kserve-8f555f6bb-qzxhq\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:47.144282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:47.144181 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:47.442114 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:47.442088 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq"] Oct 30 18:10:47.443322 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:10:47.443296 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8673e2ff_31e6_4172_ab13_8e5ea74713cf.slice/crio-ef0a385512aa346c6c3cc93fd690af087445993ad50c878abfc96cab5ea50485 WatchSource:0}: Error finding container ef0a385512aa346c6c3cc93fd690af087445993ad50c878abfc96cab5ea50485: Status 404 returned error can't find the container with id ef0a385512aa346c6c3cc93fd690af087445993ad50c878abfc96cab5ea50485 Oct 30 18:10:47.485666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:47.485645 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:47.491538 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:47.491419 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" event={"ID":"8673e2ff-31e6-4172-ab13-8e5ea74713cf","Type":"ContainerStarted","Data":"ef0a385512aa346c6c3cc93fd690af087445993ad50c878abfc96cab5ea50485"} Oct 30 18:10:47.501190 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:47.501170 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:48.497796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:48.497763 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" event={"ID":"8673e2ff-31e6-4172-ab13-8e5ea74713cf","Type":"ContainerStarted","Data":"d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2"} Oct 30 18:10:48.498977 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:48.498955 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:10:48.538603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:48.538560 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podStartSLOduration=2.538547422 podStartE2EDuration="2.538547422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:48.537398495 +0000 UTC m=+747.586443418" watchObservedRunningTime="2026-10-30 18:10:48.538547422 +0000 UTC m=+747.587592334" Oct 30 18:10:49.144490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:49.144461 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:49.508340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:49.507546 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:10:49.596440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:49.596407 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:49.610068 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:49.610040 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:49.872330 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:10:49.872222 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:10:50.520611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:50.520584 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:10:54.741830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.741794 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z"] Oct 30 18:10:54.746881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.746858 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.770008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.769942 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z"] Oct 30 18:10:54.882649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.882621 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.882803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.882654 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.882803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.882694 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d13ba25-0745-413d-8667-3edb42af0102-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.882939 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.882814 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d13ba25-0745-413d-8667-3edb42af0102-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.882939 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.882875 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.883039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.882948 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.883039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.883003 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.883106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.883049 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttm7\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-kube-api-access-cttm7\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.883106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.883072 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984117 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984087 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984131 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984187 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d13ba25-0745-413d-8667-3edb42af0102-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984215 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d13ba25-0745-413d-8667-3edb42af0102-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984284 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984349 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984380 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984415 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttm7\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-kube-api-access-cttm7\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984445 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984677 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984578 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984740 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.984986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.984954 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d13ba25-0745-413d-8667-3edb42af0102-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.985062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.985011 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.985170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.985147 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.986895 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.986874 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d13ba25-0745-413d-8667-3edb42af0102-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:54.987133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:54.987115 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:55.003077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.003005 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttm7\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-kube-api-access-cttm7\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:55.003458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.003440 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:55.062029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.062006 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:55.378005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.377980 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z"] Oct 30 18:10:55.379938 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:10:55.379913 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d13ba25_0745_413d_8667_3edb42af0102.slice/crio-4a885bba1ec69325754a253280dffb97a1e29fda3028f5837ed3b943b90b20f7 WatchSource:0}: Error finding container 4a885bba1ec69325754a253280dffb97a1e29fda3028f5837ed3b943b90b20f7: Status 404 returned error can't find the container with id 4a885bba1ec69325754a253280dffb97a1e29fda3028f5837ed3b943b90b20f7 Oct 30 18:10:55.382325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.382294 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:55.382408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.382367 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:55.382408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.382399 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:10:55.535708 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.535670 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" event={"ID":"3d13ba25-0745-413d-8667-3edb42af0102","Type":"ContainerStarted","Data":"0c73a1b1b2084d38ff6f01c04a5bbc275cadfa147b14e51b3cffb1df6fe52a2a"} Oct 30 18:10:55.535846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.535729 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" event={"ID":"3d13ba25-0745-413d-8667-3edb42af0102","Type":"ContainerStarted","Data":"4a885bba1ec69325754a253280dffb97a1e29fda3028f5837ed3b943b90b20f7"} Oct 30 18:10:55.576083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:55.575776 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" podStartSLOduration=1.575761269 podStartE2EDuration="1.575761269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:55.575674142 +0000 UTC m=+754.624719054" watchObservedRunningTime="2026-10-30 18:10:55.575761269 +0000 UTC m=+754.624806184" Oct 30 18:10:56.062737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:56.062703 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:10:56.064349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:56.064323 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:56.064471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:56.064370 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:10:57.062852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:57.062817 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:57.063285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:57.062869 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:10:57.145665 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:57.145639 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:57.166226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:57.166199 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:57.579280 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:57.579243 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:10:58.063306 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.063279 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:58.063687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.063327 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:10:58.217442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.217409 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw"] Oct 30 18:10:58.220933 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.220913 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.259713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.259687 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw"] Oct 30 18:10:58.269933 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.269906 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"kube-root-ca.crt\"" Oct 30 18:10:58.270104 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.269946 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"openshift-service-ca.crt\"" Oct 30 18:10:58.270104 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.269996 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"default-dockercfg-jzhmp\"" Oct 30 18:10:58.270254 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.270005 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"simulator-w4-fe1374-kserve-self-signed-certs\"" Oct 30 18:10:58.321462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.321384 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdv2s\" (UniqueName: \"kubernetes.io/projected/f0670173-6c92-4b83-913f-946d879df4a1-kube-api-access-wdv2s\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.321462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.321432 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-model-cache\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.321462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.321457 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0670173-6c92-4b83-913f-946d879df4a1-tls-certs\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.321680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.321641 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-home\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.321736 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.321720 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-dshm\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.321863 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.321842 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-tmp-dir\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.422558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.422526 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-model-cache\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.422761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.422578 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0670173-6c92-4b83-913f-946d879df4a1-tls-certs\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.422761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.422639 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-home\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.422761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.422699 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-dshm\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.422932 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.422780 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-tmp-dir\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.422932 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.422810 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdv2s\" (UniqueName: \"kubernetes.io/projected/f0670173-6c92-4b83-913f-946d879df4a1-kube-api-access-wdv2s\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.423038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.422962 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-model-cache\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.423038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.423000 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-home\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.423245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.423225 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-tmp-dir\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.425159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.425134 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-dshm\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.425405 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.425388 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0670173-6c92-4b83-913f-946d879df4a1-tls-certs\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.461358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.461232 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdv2s\" (UniqueName: \"kubernetes.io/projected/f0670173-6c92-4b83-913f-946d879df4a1-kube-api-access-wdv2s\") pod \"simulator-w4-fe1374-kserve-784fb8dd45-95qxw\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.564368 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.564339 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:10:58.869239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:58.869209 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw"] Oct 30 18:10:58.870532 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:10:58.870504 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0670173_6c92_4b83_913f_946d879df4a1.slice/crio-0aafa6771c89f23bbb595c99724d5fe029ca7ea1b69ecf051cbd6199be19d933 WatchSource:0}: Error finding container 0aafa6771c89f23bbb595c99724d5fe029ca7ea1b69ecf051cbd6199be19d933: Status 404 returned error can't find the container with id 0aafa6771c89f23bbb595c99724d5fe029ca7ea1b69ecf051cbd6199be19d933 Oct 30 18:10:59.062337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:59.062310 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:10:59.062496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:59.062362 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:10:59.573546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:59.573507 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" event={"ID":"f0670173-6c92-4b83-913f-946d879df4a1","Type":"ContainerStarted","Data":"f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b"} Oct 30 18:10:59.574021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:59.573558 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" event={"ID":"f0670173-6c92-4b83-913f-946d879df4a1","Type":"ContainerStarted","Data":"0aafa6771c89f23bbb595c99724d5fe029ca7ea1b69ecf051cbd6199be19d933"} Oct 30 18:10:59.615573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:10:59.615530 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podStartSLOduration=1.615516497 podStartE2EDuration="1.615516497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:59.612623944 +0000 UTC m=+758.661668857" watchObservedRunningTime="2026-10-30 18:10:59.615516497 +0000 UTC m=+758.664561462" Oct 30 18:11:00.062481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:00.062448 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:00.062653 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:00.062502 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:11:00.565369 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:00.565337 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:11:01.063196 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:01.063166 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:01.063589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:01.063216 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 18:11:02.066622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:02.066597 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:11:02.067060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:02.067048 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:11:02.067831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:02.067811 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:11:04.073754 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:04.073717 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:11:08.565388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:08.565355 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:11:08.578409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:08.578380 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:11:08.637277 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:08.637234 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:11:09.759275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.759240 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj"] Oct 30 18:11:09.763110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.763091 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.770715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.770613 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.770715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.770647 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.770845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.770742 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.770845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.770783 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxnmz\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-kube-api-access-mxnmz\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.770845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.770804 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.770946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.770849 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.770946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.770893 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.770946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.770926 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.771037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.770949 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.793883 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.793856 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj"] Oct 30 18:11:09.871515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871481 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.871687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871526 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.871687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871566 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.871687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871594 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxnmz\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-kube-api-access-mxnmz\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.871687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871620 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.871687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871673 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.871958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871733 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.871958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871786 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.871958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871819 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.871958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871906 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-data\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.872187 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.871954 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-credential-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.872187 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.872149 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-certs\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.872351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.872324 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-socket\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.872590 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.872571 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istiod-ca-cert\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.874504 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.874474 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-podinfo\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.874605 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.874570 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-envoy\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.890586 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.890555 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-token\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:09.890803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:09.890781 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxnmz\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-kube-api-access-mxnmz\") pod \"e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:10.077549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:10.077517 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:10.394213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:10.394190 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj"] Oct 30 18:11:10.396359 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:11:10.396336 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6294d21_b8ea_4c03_8d37_9abc5e722b51.slice/crio-f21dd93dacaf3b2bddc70f9157719a22d9271f7cedacab70a1fa6640f3a239c0 WatchSource:0}: Error finding container f21dd93dacaf3b2bddc70f9157719a22d9271f7cedacab70a1fa6640f3a239c0: Status 404 returned error can't find the container with id f21dd93dacaf3b2bddc70f9157719a22d9271f7cedacab70a1fa6640f3a239c0 Oct 30 18:11:10.398653 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:10.398625 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:10.398733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:10.398688 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:10.398733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:10.398728 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:11:10.637773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:10.637742 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" event={"ID":"e6294d21-b8ea-4c03-8d37-9abc5e722b51","Type":"ContainerStarted","Data":"a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f"} Oct 30 18:11:10.637962 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:10.637792 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" event={"ID":"e6294d21-b8ea-4c03-8d37-9abc5e722b51","Type":"ContainerStarted","Data":"f21dd93dacaf3b2bddc70f9157719a22d9271f7cedacab70a1fa6640f3a239c0"} Oct 30 18:11:10.680478 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:10.680375 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" podStartSLOduration=1.6803619840000001 podStartE2EDuration="1.680361984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:10.679070661 +0000 UTC m=+769.728115573" watchObservedRunningTime="2026-10-30 18:11:10.680361984 +0000 UTC m=+769.729406960" Oct 30 18:11:11.077790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:11.077762 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:11.079230 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:11.079209 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:11.079357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:11.079251 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:11:12.078669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:12.078632 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:12.079062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:12.078691 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:11:13.078849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:13.078815 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:13.079356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:13.078871 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:11:14.078070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:14.078030 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:14.078226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:14.078108 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:11:15.078871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:15.078838 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:15.079287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:15.078896 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:11:16.078839 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:16.078803 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:16.079013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:16.078863 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 18:11:16.436518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:16.436444 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:11:17.354388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:17.354352 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:17.355672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:17.355650 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:17.356577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:17.356555 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 18:11:18.410588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.410557 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx"] Oct 30 18:11:18.415785 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.415742 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.438703 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.438676 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx"] Oct 30 18:11:18.446416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.446394 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-380c08\"/\"premium-w1-380c08-kserve-self-signed-certs\"" Oct 30 18:11:18.503112 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.503083 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bclfk\" (UniqueName: \"kubernetes.io/projected/40fd49b0-06e5-4b9a-9489-ee3682f37a09-kube-api-access-bclfk\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.503300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.503120 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-dshm\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.503300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.503138 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-home\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.503300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.503216 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tmp-dir\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.503446 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.503398 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tls-certs\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.503446 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.503426 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-model-cache\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.605901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.605861 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-home\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.606645 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.606621 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tmp-dir\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.607181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.606520 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-home\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.607347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.607205 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tls-certs\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.607422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.607055 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tmp-dir\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.607484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.607466 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-model-cache\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.607587 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.607565 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bclfk\" (UniqueName: \"kubernetes.io/projected/40fd49b0-06e5-4b9a-9489-ee3682f37a09-kube-api-access-bclfk\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.607774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.607746 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-dshm\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.607915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.607897 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-model-cache\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.610540 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.610511 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-dshm\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.610640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.610618 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tls-certs\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.627986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.627955 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bclfk\" (UniqueName: \"kubernetes.io/projected/40fd49b0-06e5-4b9a-9489-ee3682f37a09-kube-api-access-bclfk\") pod \"premium-w1-380c08-kserve-7f9bcfbdc-rn8lx\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:18.734176 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:18.734099 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:19.031385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:19.031360 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx"] Oct 30 18:11:19.032961 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:11:19.032936 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40fd49b0_06e5_4b9a_9489_ee3682f37a09.slice/crio-7ffa3870e35ccbf36ec4d3c0f302b3a14355a7cf9ee8ec8b6aa0b3d4d3b639ba WatchSource:0}: Error finding container 7ffa3870e35ccbf36ec4d3c0f302b3a14355a7cf9ee8ec8b6aa0b3d4d3b639ba: Status 404 returned error can't find the container with id 7ffa3870e35ccbf36ec4d3c0f302b3a14355a7cf9ee8ec8b6aa0b3d4d3b639ba Oct 30 18:11:19.700372 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:19.700341 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" event={"ID":"40fd49b0-06e5-4b9a-9489-ee3682f37a09","Type":"ContainerStarted","Data":"0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82"} Oct 30 18:11:19.700372 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:19.700378 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" event={"ID":"40fd49b0-06e5-4b9a-9489-ee3682f37a09","Type":"ContainerStarted","Data":"7ffa3870e35ccbf36ec4d3c0f302b3a14355a7cf9ee8ec8b6aa0b3d4d3b639ba"} Oct 30 18:11:19.734967 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:19.734923 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:19.740984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:19.740934 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podStartSLOduration=1.74092001 podStartE2EDuration="1.74092001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:19.738051413 +0000 UTC m=+778.787096327" watchObservedRunningTime="2026-10-30 18:11:19.74092001 +0000 UTC m=+778.789964923" Oct 30 18:11:19.877488 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:11:19.877447 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:11:28.735078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:28.735044 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:28.747823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:28.747799 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:28.842532 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:28.842506 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:11:31.344218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:31.344184 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:11:48.331756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:11:48.331722 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:11:49.886129 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:11:49.886093 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:12:19.870379 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:12:19.870342 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:12:34.578550 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.577311 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw"] Oct 30 18:12:34.584867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.584847 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.599580 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.599556 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw"] Oct 30 18:12:34.608540 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.608508 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"e2e-conflict-a-cd0ee102-kserve-self-signed-certs\"" Oct 30 18:12:34.680469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.680441 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-dshm\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.680585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.680528 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncf22\" (UniqueName: \"kubernetes.io/projected/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-kube-api-access-ncf22\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.680585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.680575 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-home\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.680743 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.680722 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tls-certs\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.680895 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.680869 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-model-cache\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.680968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.680952 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tmp-dir\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.782422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.782385 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-model-cache\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.782620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.782437 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tmp-dir\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.782620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.782486 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-dshm\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.782620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.782534 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncf22\" (UniqueName: \"kubernetes.io/projected/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-kube-api-access-ncf22\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.782620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.782569 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-home\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.782834 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.782624 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tls-certs\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.782892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.782857 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-model-cache\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.782972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.782947 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tmp-dir\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.783160 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.783131 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-home\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.784994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.784970 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-dshm\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.785310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.785283 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tls-certs\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.803186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.803158 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncf22\" (UniqueName: \"kubernetes.io/projected/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-kube-api-access-ncf22\") pod \"e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:34.924014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:34.923935 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:35.223110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:35.222980 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw"] Oct 30 18:12:35.226122 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:12:35.226086 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bb32e86_4ccb_44ab_8662_a5b6917ece5a.slice/crio-501b390a5292ba95547bf5da59db4599201b71fb3c2f78caeb8ca72858e8db2e WatchSource:0}: Error finding container 501b390a5292ba95547bf5da59db4599201b71fb3c2f78caeb8ca72858e8db2e: Status 404 returned error can't find the container with id 501b390a5292ba95547bf5da59db4599201b71fb3c2f78caeb8ca72858e8db2e Oct 30 18:12:36.194742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:36.194704 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" event={"ID":"9bb32e86-4ccb-44ab-8662-a5b6917ece5a","Type":"ContainerStarted","Data":"dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048"} Oct 30 18:12:36.194742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:36.194750 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" event={"ID":"9bb32e86-4ccb-44ab-8662-a5b6917ece5a","Type":"ContainerStarted","Data":"501b390a5292ba95547bf5da59db4599201b71fb3c2f78caeb8ca72858e8db2e"} Oct 30 18:12:36.239591 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:36.239546 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" podStartSLOduration=2.2395312450000002 podStartE2EDuration="2.239531245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:36.236404345 +0000 UTC m=+855.285449259" watchObservedRunningTime="2026-10-30 18:12:36.239531245 +0000 UTC m=+855.288576158" Oct 30 18:12:36.924837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:36.924808 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:38.520255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:38.520196 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:12:44.924347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:44.924311 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:44.937691 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:44.937664 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:45.253377 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:45.253316 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:12:49.869157 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:12:49.869110 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:12:51.408053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.408021 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p"] Oct 30 18:12:51.411993 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.411970 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.441184 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.441161 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"e2e-conflict-b-cd0ee102-kserve-self-signed-certs\"" Oct 30 18:12:51.444837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.444812 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p"] Oct 30 18:12:51.581434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.579861 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tls-certs\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.581434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.579913 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-home\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.581434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.579939 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-dshm\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.581434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.580029 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tmp-dir\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.581434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.580112 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-model-cache\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.581434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.580163 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sssg\" (UniqueName: \"kubernetes.io/projected/a9cf22a5-5166-4a2d-a457-7ae4664ff388-kube-api-access-9sssg\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.681718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.681620 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tmp-dir\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.681898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.681735 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-model-cache\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.681898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.681794 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sssg\" (UniqueName: \"kubernetes.io/projected/a9cf22a5-5166-4a2d-a457-7ae4664ff388-kube-api-access-9sssg\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.681898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.681849 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tls-certs\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.681898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.681885 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-home\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.682118 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.681909 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-dshm\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.682176 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.682136 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tmp-dir\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.682626 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.682596 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-model-cache\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.682790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.682642 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-home\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.684918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.684892 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-dshm\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.685661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.685637 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tls-certs\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.702193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.702160 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sssg\" (UniqueName: \"kubernetes.io/projected/a9cf22a5-5166-4a2d-a457-7ae4664ff388-kube-api-access-9sssg\") pod \"e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:51.733291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:51.732468 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:52.041536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:52.041505 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p"] Oct 30 18:12:52.043247 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:12:52.043215 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9cf22a5_5166_4a2d_a457_7ae4664ff388.slice/crio-64fec9530daddd3934d1fe413e99fc843e4693a1e6414e361bf880567b5db564 WatchSource:0}: Error finding container 64fec9530daddd3934d1fe413e99fc843e4693a1e6414e361bf880567b5db564: Status 404 returned error can't find the container with id 64fec9530daddd3934d1fe413e99fc843e4693a1e6414e361bf880567b5db564 Oct 30 18:12:52.279333 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:52.279294 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" event={"ID":"a9cf22a5-5166-4a2d-a457-7ae4664ff388","Type":"ContainerStarted","Data":"7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652"} Oct 30 18:12:52.279497 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:52.279347 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" event={"ID":"a9cf22a5-5166-4a2d-a457-7ae4664ff388","Type":"ContainerStarted","Data":"64fec9530daddd3934d1fe413e99fc843e4693a1e6414e361bf880567b5db564"} Oct 30 18:12:52.324765 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:52.324722 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" podStartSLOduration=1.324708608 podStartE2EDuration="1.324708608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:52.323672977 +0000 UTC m=+871.372717890" watchObservedRunningTime="2026-10-30 18:12:52.324708608 +0000 UTC m=+871.373753521" Oct 30 18:12:52.734448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:52.734374 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:12:56.638626 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:12:56.638596 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:13:01.733721 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:01.733682 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:13:01.746601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:01.746573 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:13:02.347131 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:02.347099 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:13:11.554376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.554344 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw"] Oct 30 18:13:11.558616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.558600 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.587684 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.587664 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fe1374-openshift-default-dockercfg-dg2jw\"" Oct 30 18:13:11.589960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.589938 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw"] Oct 30 18:13:11.619533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.619507 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.619631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.619543 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.619631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.619570 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.619705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.619674 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.619761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.619746 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4bce15-f17a-460c-acb5-b05073ec9eda-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.619797 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.619780 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.619832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.619796 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.619881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.619862 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.620006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.619990 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qhfv\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-kube-api-access-8qhfv\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.720985 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721016 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721032 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721051 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721106 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4bce15-f17a-460c-acb5-b05073ec9eda-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721129 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721143 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721191 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721314 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qhfv\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-kube-api-access-8qhfv\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721624 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721490 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721624 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721550 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721624 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721591 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721725 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721639 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.721940 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.721920 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4bce15-f17a-460c-acb5-b05073ec9eda-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.723755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.723732 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.723856 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.723770 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.739969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.739936 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.740053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.740017 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qhfv\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-kube-api-access-8qhfv\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:11.872677 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:11.872622 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:12.191825 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:12.191800 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw"] Oct 30 18:13:12.193977 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:13:12.193942 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4bce15_f17a_460c_acb5_b05073ec9eda.slice/crio-ddee0906d25b1f994538193ceb46fea9fb9abe222dfec2f50264b8ea620c644f WatchSource:0}: Error finding container ddee0906d25b1f994538193ceb46fea9fb9abe222dfec2f50264b8ea620c644f: Status 404 returned error can't find the container with id ddee0906d25b1f994538193ceb46fea9fb9abe222dfec2f50264b8ea620c644f Oct 30 18:13:12.196217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:12.196188 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:12.196309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:12.196249 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:12.196309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:12.196294 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:13:12.389800 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:12.389768 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" event={"ID":"0c4bce15-f17a-460c-acb5-b05073ec9eda","Type":"ContainerStarted","Data":"b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089"} Oct 30 18:13:12.389995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:12.389817 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" event={"ID":"0c4bce15-f17a-460c-acb5-b05073ec9eda","Type":"ContainerStarted","Data":"ddee0906d25b1f994538193ceb46fea9fb9abe222dfec2f50264b8ea620c644f"} Oct 30 18:13:12.431518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:12.431462 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" podStartSLOduration=1.431445295 podStartE2EDuration="1.431445295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:12.431188243 +0000 UTC m=+891.480233156" watchObservedRunningTime="2026-10-30 18:13:12.431445295 +0000 UTC m=+891.480490209" Oct 30 18:13:12.873313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:12.873279 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:13.778699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:13.776866 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p"] Oct 30 18:13:13.874630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:13.874547 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:13:13.875212 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:13.875172 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:13:14.725059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:14.725019 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:13:14.873310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:14.873253 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:13:14.873483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:14.873332 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:13:15.873303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:15.873250 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:13:15.873717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:15.873338 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:13:16.873888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:16.873854 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:13:16.874480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:16.873918 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:13:17.873204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:17.873175 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:13:17.873409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:17.873234 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:13:18.873826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:18.873797 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:13:18.874382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:18.873854 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 18:13:19.875534 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:13:19.875496 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:13:20.149243 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:20.149135 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:20.149635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:20.149594 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:20.150396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:20.150370 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:13:20.826567 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:20.826535 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg"] Oct 30 18:13:20.826751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:20.826729 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" containerID="cri-o://e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56" gracePeriod=30 Oct 30 18:13:21.146217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:21.146149 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:13:21.147125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:21.147107 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:13:25.161614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:25.161573 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw"] Oct 30 18:13:26.081740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.081715 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:13:26.201744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.201668 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-kube-api-access-gsw2l\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-kube-api-access-gsw2l\") pod \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " Oct 30 18:13:26.201744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.201718 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-token\") pod \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " Oct 30 18:13:26.202256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.201762 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-podinfo\") pod \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " Oct 30 18:13:26.202256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.201831 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-socket\") pod \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " Oct 30 18:13:26.202256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.201879 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istiod-ca-cert\") pod \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " Oct 30 18:13:26.202256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.201903 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-certs\") pod \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " Oct 30 18:13:26.202256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.201941 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-data\") pod \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " Oct 30 18:13:26.202256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202003 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-credential-socket\") pod \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " Oct 30 18:13:26.202256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202050 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-envoy\") pod \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\" (UID: \"3a5b64a1-2f6b-4120-ab40-76b3bc209992\") " Oct 30 18:13:26.202256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202216 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-socket" pod "3a5b64a1-2f6b-4120-ab40-76b3bc209992" (UID: "3a5b64a1-2f6b-4120-ab40-76b3bc209992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:26.202897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202370 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istiod-ca-cert" pod "3a5b64a1-2f6b-4120-ab40-76b3bc209992" (UID: "3a5b64a1-2f6b-4120-ab40-76b3bc209992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:26.202897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202467 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-data" pod "3a5b64a1-2f6b-4120-ab40-76b3bc209992" (UID: "3a5b64a1-2f6b-4120-ab40-76b3bc209992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:26.202897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202593 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-credential-socket" pod "3a5b64a1-2f6b-4120-ab40-76b3bc209992" (UID: "3a5b64a1-2f6b-4120-ab40-76b3bc209992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:26.202897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202687 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-certs" pod "3a5b64a1-2f6b-4120-ab40-76b3bc209992" (UID: "3a5b64a1-2f6b-4120-ab40-76b3bc209992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:26.202897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202867 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:26.202897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202893 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:26.203222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202909 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:26.203222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202924 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:26.203222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.202937 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:26.204098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.204077 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-kube-api-access-gsw2l" pod "3a5b64a1-2f6b-4120-ab40-76b3bc209992" (UID: "3a5b64a1-2f6b-4120-ab40-76b3bc209992"). InnerVolumeSpecName "kube-api-access-gsw2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:26.204452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.204350 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-podinfo" pod "3a5b64a1-2f6b-4120-ab40-76b3bc209992" (UID: "3a5b64a1-2f6b-4120-ab40-76b3bc209992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:13:26.204452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.204379 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-token" pod "3a5b64a1-2f6b-4120-ab40-76b3bc209992" (UID: "3a5b64a1-2f6b-4120-ab40-76b3bc209992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:26.204602 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.204511 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-envoy" pod "3a5b64a1-2f6b-4120-ab40-76b3bc209992" (UID: "3a5b64a1-2f6b-4120-ab40-76b3bc209992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:26.304181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.304154 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:26.304181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.304182 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsw2l\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-kube-api-access-gsw2l\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:26.304396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.304196 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:26.304396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.304207 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5b64a1-2f6b-4120-ab40-76b3bc209992-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:26.512061 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.511988 4862 generic.go:356] "Generic (PLEG): container finished" podID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerID="e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56" exitCode=0 Oct 30 18:13:26.512061 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.512033 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" Oct 30 18:13:26.512273 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.512062 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" event={"ID":"3a5b64a1-2f6b-4120-ab40-76b3bc209992","Type":"ContainerDied","Data":"e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56"} Oct 30 18:13:26.512273 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.512144 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg" event={"ID":"3a5b64a1-2f6b-4120-ab40-76b3bc209992","Type":"ContainerDied","Data":"3f9a055ef82536a9acfb99ba2bfd195e0790628c8107285e81a65bbf12dac6a4"} Oct 30 18:13:26.512273 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.512167 4862 scope.go:122] "RemoveContainer" containerID="e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56" Oct 30 18:13:26.522957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.522941 4862 scope.go:122] "RemoveContainer" containerID="e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56" Oct 30 18:13:26.523228 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:13:26.523209 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56\": container with ID starting with e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56 not found: ID does not exist" containerID="e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56" Oct 30 18:13:26.523320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.523236 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56"} err="failed to get container status \"e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56\": rpc error: code = NotFound desc = could not find container \"e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56\": container with ID starting with e0608873844d155b016fe20c2a941493af2cb16b518054be029b9e49135e5d56 not found: ID does not exist" Oct 30 18:13:26.563934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.563912 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg"] Oct 30 18:13:26.576666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:26.576645 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-6qqzg"] Oct 30 18:13:27.061092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:27.061058 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" path="/var/lib/kubelet/pods/3a5b64a1-2f6b-4120-ab40-76b3bc209992/volumes" Oct 30 18:13:28.811150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:28.811100 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" podUID="a9cf22a5-5166-4a2d-a457-7ae4664ff388" containerName="main" containerID="cri-o://7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652" gracePeriod=45 Oct 30 18:13:29.086459 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.086436 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:13:29.236838 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.236811 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tls-certs\") pod \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " Oct 30 18:13:29.237027 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.236857 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-home\") pod \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " Oct 30 18:13:29.237027 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.236910 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-model-cache\") pod \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " Oct 30 18:13:29.237027 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.236949 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-dshm\") pod \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " Oct 30 18:13:29.237202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.237042 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cf22a5-5166-4a2d-a457-7ae4664ff388-kube-api-access-9sssg\" (UniqueName: \"kubernetes.io/projected/a9cf22a5-5166-4a2d-a457-7ae4664ff388-kube-api-access-9sssg\") pod \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " Oct 30 18:13:29.237202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.237116 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tmp-dir\") pod \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\" (UID: \"a9cf22a5-5166-4a2d-a457-7ae4664ff388\") " Oct 30 18:13:29.237202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.237168 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-home" pod "a9cf22a5-5166-4a2d-a457-7ae4664ff388" (UID: "a9cf22a5-5166-4a2d-a457-7ae4664ff388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:29.237202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.237185 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-model-cache" pod "a9cf22a5-5166-4a2d-a457-7ae4664ff388" (UID: "a9cf22a5-5166-4a2d-a457-7ae4664ff388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:29.237477 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.237455 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tmp-dir" pod "a9cf22a5-5166-4a2d-a457-7ae4664ff388" (UID: "a9cf22a5-5166-4a2d-a457-7ae4664ff388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:29.237639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.237622 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:29.237723 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.237644 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:29.237723 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.237657 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:29.239253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.239234 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-dshm" pod "a9cf22a5-5166-4a2d-a457-7ae4664ff388" (UID: "a9cf22a5-5166-4a2d-a457-7ae4664ff388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:29.239783 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.239763 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tls-certs" pod "a9cf22a5-5166-4a2d-a457-7ae4664ff388" (UID: "a9cf22a5-5166-4a2d-a457-7ae4664ff388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:29.239879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.239788 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cf22a5-5166-4a2d-a457-7ae4664ff388-kube-api-access-9sssg" pod "a9cf22a5-5166-4a2d-a457-7ae4664ff388" (UID: "a9cf22a5-5166-4a2d-a457-7ae4664ff388"). InnerVolumeSpecName "kube-api-access-9sssg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:29.338823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.338753 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9cf22a5-5166-4a2d-a457-7ae4664ff388-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:29.338823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.338792 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9cf22a5-5166-4a2d-a457-7ae4664ff388-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:29.338823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.338803 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sssg\" (UniqueName: \"kubernetes.io/projected/a9cf22a5-5166-4a2d-a457-7ae4664ff388-kube-api-access-9sssg\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:29.536586 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.536552 4862 generic.go:356] "Generic (PLEG): container finished" podID="a9cf22a5-5166-4a2d-a457-7ae4664ff388" containerID="7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652" exitCode=0 Oct 30 18:13:29.536745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.536599 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" Oct 30 18:13:29.536745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.536635 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" event={"ID":"a9cf22a5-5166-4a2d-a457-7ae4664ff388","Type":"ContainerDied","Data":"7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652"} Oct 30 18:13:29.536745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.536687 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p" event={"ID":"a9cf22a5-5166-4a2d-a457-7ae4664ff388","Type":"ContainerDied","Data":"64fec9530daddd3934d1fe413e99fc843e4693a1e6414e361bf880567b5db564"} Oct 30 18:13:29.536745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.536709 4862 scope.go:122] "RemoveContainer" containerID="7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652" Oct 30 18:13:29.547614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.547597 4862 scope.go:122] "RemoveContainer" containerID="7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652" Oct 30 18:13:29.547852 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:13:29.547833 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652\": container with ID starting with 7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652 not found: ID does not exist" containerID="7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652" Oct 30 18:13:29.547918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.547859 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652"} err="failed to get container status \"7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652\": rpc error: code = NotFound desc = could not find container \"7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652\": container with ID starting with 7d6aaacbc7a77aa3f1c1aefa24a6676c8ae9d99382c2dce2fbb05847e1daa652 not found: ID does not exist" Oct 30 18:13:29.587336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.587309 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p"] Oct 30 18:13:29.599609 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:29.599558 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-b-cd0ee102-kserve-6bd876c6d5-zp58p"] Oct 30 18:13:30.574955 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:30.574929 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:13:31.089290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.076564 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cf22a5-5166-4a2d-a457-7ae4664ff388" path="/var/lib/kubelet/pods/a9cf22a5-5166-4a2d-a457-7ae4664ff388/volumes" Oct 30 18:13:31.749794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.749758 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6"] Oct 30 18:13:31.750479 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.750459 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" Oct 30 18:13:31.750533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.750483 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" Oct 30 18:13:31.750579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.750548 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a9cf22a5-5166-4a2d-a457-7ae4664ff388" containerName="main" Oct 30 18:13:31.750579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.750556 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cf22a5-5166-4a2d-a457-7ae4664ff388" containerName="main" Oct 30 18:13:31.750707 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.750694 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cf22a5-5166-4a2d-a457-7ae4664ff388" containerName="main" Oct 30 18:13:31.750741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.750713 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5b64a1-2f6b-4120-ab40-76b3bc209992" containerName="istio-proxy" Oct 30 18:13:31.755839 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.755815 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.773409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.773385 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6"] Oct 30 18:13:31.783730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.783707 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-de7e3a\"/\"distinct-w3-de7e3a-kserve-self-signed-certs\"" Oct 30 18:13:31.871954 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.871923 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnkv\" (UniqueName: \"kubernetes.io/projected/3fc89dcc-a44d-477a-9570-6f5a7aad5357-kube-api-access-lbnkv\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.872149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.871969 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-model-cache\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.872149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.872042 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tls-certs\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.872149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.872085 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-dshm\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.872356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.872156 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-home\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.872356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.872219 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tmp-dir\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.973584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.973544 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnkv\" (UniqueName: \"kubernetes.io/projected/3fc89dcc-a44d-477a-9570-6f5a7aad5357-kube-api-access-lbnkv\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.973772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.973598 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-model-cache\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.973772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.973646 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tls-certs\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.973772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.973712 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-dshm\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.973944 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.973771 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-home\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.973944 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.973830 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tmp-dir\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.974090 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.974067 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-model-cache\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.974217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.974195 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-home\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.974308 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.974215 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tmp-dir\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.976086 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.976065 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-dshm\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.976835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.976813 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tls-certs\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:31.995764 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:31.995724 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnkv\" (UniqueName: \"kubernetes.io/projected/3fc89dcc-a44d-477a-9570-6f5a7aad5357-kube-api-access-lbnkv\") pod \"distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:32.067571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:32.067543 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:32.369103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:32.369081 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6"] Oct 30 18:13:32.370989 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:13:32.370965 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fc89dcc_a44d_477a_9570_6f5a7aad5357.slice/crio-9296e502b28f9848bec43fb3096cd75d6c3a66ca4b9ece61c8eb262f12bbf7b1 WatchSource:0}: Error finding container 9296e502b28f9848bec43fb3096cd75d6c3a66ca4b9ece61c8eb262f12bbf7b1: Status 404 returned error can't find the container with id 9296e502b28f9848bec43fb3096cd75d6c3a66ca4b9ece61c8eb262f12bbf7b1 Oct 30 18:13:32.556688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:32.556646 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" event={"ID":"3fc89dcc-a44d-477a-9570-6f5a7aad5357","Type":"ContainerStarted","Data":"d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0"} Oct 30 18:13:32.556872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:32.556696 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" event={"ID":"3fc89dcc-a44d-477a-9570-6f5a7aad5357","Type":"ContainerStarted","Data":"9296e502b28f9848bec43fb3096cd75d6c3a66ca4b9ece61c8eb262f12bbf7b1"} Oct 30 18:13:32.608116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:32.608068 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podStartSLOduration=1.608056446 podStartE2EDuration="1.608056446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:32.605893875 +0000 UTC m=+911.654938788" watchObservedRunningTime="2026-10-30 18:13:32.608056446 +0000 UTC m=+911.657101361" Oct 30 18:13:33.068078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:33.068053 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:40.183388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.183348 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" podUID="9bb32e86-4ccb-44ab-8662-a5b6917ece5a" containerName="main" containerID="cri-o://dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048" gracePeriod=45 Oct 30 18:13:40.243311 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.243282 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" podUID="9bb32e86-4ccb-44ab-8662-a5b6917ece5a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused" Oct 30 18:13:40.450027 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.449969 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:13:40.460198 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460164 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-model-cache\") pod \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " Oct 30 18:13:40.460318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460227 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-home\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-home\") pod \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " Oct 30 18:13:40.460318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460298 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tmp-dir\") pod \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " Oct 30 18:13:40.460403 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460348 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-dshm\") pod \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " Oct 30 18:13:40.460463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460447 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-kube-api-access-ncf22\" (UniqueName: \"kubernetes.io/projected/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-kube-api-access-ncf22\") pod \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " Oct 30 18:13:40.460525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460459 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-home" pod "9bb32e86-4ccb-44ab-8662-a5b6917ece5a" (UID: "9bb32e86-4ccb-44ab-8662-a5b6917ece5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:40.460525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460472 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-model-cache" pod "9bb32e86-4ccb-44ab-8662-a5b6917ece5a" (UID: "9bb32e86-4ccb-44ab-8662-a5b6917ece5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:40.460525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460501 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tls-certs\") pod \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\" (UID: \"9bb32e86-4ccb-44ab-8662-a5b6917ece5a\") " Oct 30 18:13:40.460525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460508 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tmp-dir" pod "9bb32e86-4ccb-44ab-8662-a5b6917ece5a" (UID: "9bb32e86-4ccb-44ab-8662-a5b6917ece5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:40.460803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460789 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:40.460864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460804 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:40.460864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.460813 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:40.462497 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.462476 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-dshm" pod "9bb32e86-4ccb-44ab-8662-a5b6917ece5a" (UID: "9bb32e86-4ccb-44ab-8662-a5b6917ece5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:40.462690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.462666 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-kube-api-access-ncf22" pod "9bb32e86-4ccb-44ab-8662-a5b6917ece5a" (UID: "9bb32e86-4ccb-44ab-8662-a5b6917ece5a"). InnerVolumeSpecName "kube-api-access-ncf22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:40.462783 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.462728 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tls-certs" pod "9bb32e86-4ccb-44ab-8662-a5b6917ece5a" (UID: "9bb32e86-4ccb-44ab-8662-a5b6917ece5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:40.562209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.562182 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:40.562209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.562209 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:40.562401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.562221 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncf22\" (UniqueName: \"kubernetes.io/projected/9bb32e86-4ccb-44ab-8662-a5b6917ece5a-kube-api-access-ncf22\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:13:40.598924 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.598894 4862 generic.go:356] "Generic (PLEG): container finished" podID="9bb32e86-4ccb-44ab-8662-a5b6917ece5a" containerID="dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048" exitCode=0 Oct 30 18:13:40.599056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.599008 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" Oct 30 18:13:40.599056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.599032 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" event={"ID":"9bb32e86-4ccb-44ab-8662-a5b6917ece5a","Type":"ContainerDied","Data":"dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048"} Oct 30 18:13:40.599139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.599080 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw" event={"ID":"9bb32e86-4ccb-44ab-8662-a5b6917ece5a","Type":"ContainerDied","Data":"501b390a5292ba95547bf5da59db4599201b71fb3c2f78caeb8ca72858e8db2e"} Oct 30 18:13:40.599139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.599098 4862 scope.go:122] "RemoveContainer" containerID="dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048" Oct 30 18:13:40.608666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.608640 4862 scope.go:122] "RemoveContainer" containerID="dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048" Oct 30 18:13:40.608971 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:13:40.608953 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048\": container with ID starting with dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048 not found: ID does not exist" containerID="dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048" Oct 30 18:13:40.609029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.608979 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048"} err="failed to get container status \"dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048\": rpc error: code = NotFound desc = could not find container \"dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048\": container with ID starting with dd2cd5e94e9ee745c7b5ef393b573ef61d0416b7f77458715a104036caf93048 not found: ID does not exist" Oct 30 18:13:40.650828 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.650810 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw"] Oct 30 18:13:40.654337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:40.654321 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/e2e-conflict-a-cd0ee102-kserve-65566469f5-kt8dw"] Oct 30 18:13:41.063139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:41.063110 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb32e86-4ccb-44ab-8662-a5b6917ece5a" path="/var/lib/kubelet/pods/9bb32e86-4ccb-44ab-8662-a5b6917ece5a/volumes" Oct 30 18:13:42.068108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:42.067909 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:42.088980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:42.088950 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:42.623064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:42.623039 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:13:44.071469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.071441 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv"] Oct 30 18:13:44.071868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.071856 4862 cpu_manager.go:414] "containerMap: removing container" podUID="9bb32e86-4ccb-44ab-8662-a5b6917ece5a" containerName="main" Oct 30 18:13:44.071868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.071867 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb32e86-4ccb-44ab-8662-a5b6917ece5a" containerName="main" Oct 30 18:13:44.071966 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.071956 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb32e86-4ccb-44ab-8662-a5b6917ece5a" containerName="main" Oct 30 18:13:44.076520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.076497 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.101917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.101886 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-dshm\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.101917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.101939 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-model-cache\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.102164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.102005 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-tmp-dir\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.102164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.102042 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-home\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.102164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.102088 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6z2\" (UniqueName: \"kubernetes.io/projected/75860e57-2ba4-449e-887c-e8d48a45d9fc-kube-api-access-hs6z2\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.102164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.102158 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75860e57-2ba4-449e-887c-e8d48a45d9fc-tls-certs\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.105547 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.105522 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"distinct-w2-45dc34-kserve-self-signed-certs\"" Oct 30 18:13:44.108431 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.108407 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv"] Oct 30 18:13:44.203751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.203705 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-tmp-dir\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.203751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.203758 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-home\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.203988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.203792 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6z2\" (UniqueName: \"kubernetes.io/projected/75860e57-2ba4-449e-887c-e8d48a45d9fc-kube-api-access-hs6z2\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.203988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.203833 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75860e57-2ba4-449e-887c-e8d48a45d9fc-tls-certs\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.203988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.203892 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-dshm\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.203988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.203922 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-model-cache\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.204232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.204205 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-home\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.204412 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.204392 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-model-cache\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.204490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.204427 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-tmp-dir\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.206505 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.206479 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-dshm\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.206693 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.206672 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75860e57-2ba4-449e-887c-e8d48a45d9fc-tls-certs\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.231390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.231362 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6z2\" (UniqueName: \"kubernetes.io/projected/75860e57-2ba4-449e-887c-e8d48a45d9fc-kube-api-access-hs6z2\") pod \"distinct-w2-45dc34-kserve-6c88f8597d-5j8zv\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.390466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.390352 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:44.693635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:44.693608 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv"] Oct 30 18:13:44.695868 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:13:44.695842 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75860e57_2ba4_449e_887c_e8d48a45d9fc.slice/crio-63cd1570410283b4d5452f9661e9bece40585f651708c84dab315dba267b68f6 WatchSource:0}: Error finding container 63cd1570410283b4d5452f9661e9bece40585f651708c84dab315dba267b68f6: Status 404 returned error can't find the container with id 63cd1570410283b4d5452f9661e9bece40585f651708c84dab315dba267b68f6 Oct 30 18:13:45.630388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:45.630352 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" event={"ID":"75860e57-2ba4-449e-887c-e8d48a45d9fc","Type":"ContainerStarted","Data":"698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95"} Oct 30 18:13:45.630788 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:45.630397 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" event={"ID":"75860e57-2ba4-449e-887c-e8d48a45d9fc","Type":"ContainerStarted","Data":"63cd1570410283b4d5452f9661e9bece40585f651708c84dab315dba267b68f6"} Oct 30 18:13:45.671822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:45.671767 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podStartSLOduration=1.671751096 podStartE2EDuration="1.671751096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:45.669720674 +0000 UTC m=+924.718765580" watchObservedRunningTime="2026-10-30 18:13:45.671751096 +0000 UTC m=+924.720796014" Oct 30 18:13:46.390858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:46.390820 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:49.872894 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:13:49.872854 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:13:50.390959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:50.390924 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:13:54.390716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.390680 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:54.408884 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.408858 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:54.717381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.717309 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:13:54.797805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.797775 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8"] Oct 30 18:13:54.803222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.803202 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:54.819563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.819539 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-921450\"/\"distinct-w0-921450-kserve-self-signed-certs\"" Oct 30 18:13:54.820294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.820254 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8"] Oct 30 18:13:54.925989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.925960 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-home\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:54.926195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.926019 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-tmp-dir\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:54.926195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.926144 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hz4h\" (UniqueName: \"kubernetes.io/projected/4bd52f6f-27c8-415c-aac0-32dad4435204-kube-api-access-4hz4h\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:54.926195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.926187 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-model-cache\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:54.926416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.926223 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-dshm\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:54.926416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:54.926357 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bd52f6f-27c8-415c-aac0-32dad4435204-tls-certs\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.028018 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.027921 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-home\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.028205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.028024 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-tmp-dir\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.028205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.028070 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hz4h\" (UniqueName: \"kubernetes.io/projected/4bd52f6f-27c8-415c-aac0-32dad4435204-kube-api-access-4hz4h\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.028205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.028100 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-model-cache\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.028205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.028132 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-dshm\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.028205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.028193 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bd52f6f-27c8-415c-aac0-32dad4435204-tls-certs\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.029430 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.029399 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-home\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.029659 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.029638 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-tmp-dir\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.029993 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.029973 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-model-cache\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.046400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.045392 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bd52f6f-27c8-415c-aac0-32dad4435204-tls-certs\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.046400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.045925 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-dshm\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.055373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.055345 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hz4h\" (UniqueName: \"kubernetes.io/projected/4bd52f6f-27c8-415c-aac0-32dad4435204-kube-api-access-4hz4h\") pod \"distinct-w0-921450-kserve-64978d4966-rhpv8\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.119059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.119029 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:13:55.417696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.417670 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8"] Oct 30 18:13:55.421169 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:13:55.421142 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bd52f6f_27c8_415c_aac0_32dad4435204.slice/crio-3d1a8b2ca54351dd8b337e0180a78be6cf01ca5c731cf889be069246ecbf51d8 WatchSource:0}: Error finding container 3d1a8b2ca54351dd8b337e0180a78be6cf01ca5c731cf889be069246ecbf51d8: Status 404 returned error can't find the container with id 3d1a8b2ca54351dd8b337e0180a78be6cf01ca5c731cf889be069246ecbf51d8 Oct 30 18:13:55.713088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.713003 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" event={"ID":"4bd52f6f-27c8-415c-aac0-32dad4435204","Type":"ContainerStarted","Data":"f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc"} Oct 30 18:13:55.713088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.713045 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" event={"ID":"4bd52f6f-27c8-415c-aac0-32dad4435204","Type":"ContainerStarted","Data":"3d1a8b2ca54351dd8b337e0180a78be6cf01ca5c731cf889be069246ecbf51d8"} Oct 30 18:13:55.754389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:55.754340 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podStartSLOduration=1.754323083 podStartE2EDuration="1.754323083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:55.752361394 +0000 UTC m=+934.801406318" watchObservedRunningTime="2026-10-30 18:13:55.754323083 +0000 UTC m=+934.803367997" Oct 30 18:13:56.119351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:13:56.119322 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:14:01.591964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.591886 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7"] Oct 30 18:14:01.595593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.595573 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.610031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.610008 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-de7e3a-openshift-default-dockercfg-p5ml9\"" Oct 30 18:14:01.626077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.625451 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7"] Oct 30 18:14:01.696289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.696235 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.696289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.696294 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.696492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.696413 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.696492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.696431 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.696492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.696447 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f6875-e361-4f37-ba52-70fa15ac77ea-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.696492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.696475 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cb95\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-kube-api-access-8cb95\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.696618 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.696604 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.696681 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.696667 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.696716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.696689 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.797563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797527 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.797563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797563 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.797806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797580 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f6875-e361-4f37-ba52-70fa15ac77ea-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.797806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797706 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cb95\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-kube-api-access-8cb95\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.797806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797769 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.797969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797857 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.797969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797887 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.797969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797941 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.798129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797970 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.798129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.797971 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.798304 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.798250 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.798467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.798409 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.798467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.798452 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.798616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.798595 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f6875-e361-4f37-ba52-70fa15ac77ea-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.800558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.800538 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.800774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.800752 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.819095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.819065 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.819408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.819390 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cb95\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-kube-api-access-8cb95\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:01.908811 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:01.908749 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:02.225957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:02.225937 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7"] Oct 30 18:14:02.228011 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:14:02.227989 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51f6875_e361_4f37_ba52_70fa15ac77ea.slice/crio-81735286fb861fe545acfa5160f406062341771f2607d8d88a8f48f93781832d WatchSource:0}: Error finding container 81735286fb861fe545acfa5160f406062341771f2607d8d88a8f48f93781832d: Status 404 returned error can't find the container with id 81735286fb861fe545acfa5160f406062341771f2607d8d88a8f48f93781832d Oct 30 18:14:02.230226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:02.230199 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:02.230328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:02.230256 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:02.230328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:02.230316 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:02.763340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:02.763309 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" event={"ID":"f51f6875-e361-4f37-ba52-70fa15ac77ea","Type":"ContainerStarted","Data":"de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5"} Oct 30 18:14:02.763765 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:02.763363 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" event={"ID":"f51f6875-e361-4f37-ba52-70fa15ac77ea","Type":"ContainerStarted","Data":"81735286fb861fe545acfa5160f406062341771f2607d8d88a8f48f93781832d"} Oct 30 18:14:02.822841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:02.822768 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" podStartSLOduration=1.822749173 podStartE2EDuration="1.822749173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:02.816061354 +0000 UTC m=+941.865106278" watchObservedRunningTime="2026-10-30 18:14:02.822749173 +0000 UTC m=+941.871794087" Oct 30 18:14:02.909368 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:02.909332 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:03.916285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:03.911501 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:14:03.916285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:03.911560 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:14:04.909493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:04.909457 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:14:04.909669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:04.909510 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:14:05.119528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:05.119495 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:14:05.156979 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:05.156952 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:14:05.791564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:05.791535 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:14:05.909383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:05.909354 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:14:05.909551 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:05.909414 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:14:06.909997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:06.909967 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:14:06.910396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:06.910030 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:14:07.909880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:07.909847 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:14:07.910039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:07.909905 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:14:09.197339 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:09.197301 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:09.197818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:09.197742 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:09.198695 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:09.198673 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:14:10.102545 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.102510 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw"] Oct 30 18:14:10.106437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.106416 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.135234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.135209 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fe1374\"/\"distinct-2-w4-fe1374-kserve-self-signed-certs\"" Oct 30 18:14:10.139086 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.139058 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw"] Oct 30 18:14:10.215866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.215833 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrldf\" (UniqueName: \"kubernetes.io/projected/d42774cd-c445-4c21-902e-6f832ec412c4-kube-api-access-rrldf\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.216315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.215966 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d42774cd-c445-4c21-902e-6f832ec412c4-tls-certs\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.216315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.216114 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-tmp-dir\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.216315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.216203 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-home\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.216315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.216231 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-dshm\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.216485 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.216395 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-model-cache\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.323026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.320959 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrldf\" (UniqueName: \"kubernetes.io/projected/d42774cd-c445-4c21-902e-6f832ec412c4-kube-api-access-rrldf\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.323026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.321038 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d42774cd-c445-4c21-902e-6f832ec412c4-tls-certs\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.323026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.321115 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-tmp-dir\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.323026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.321156 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-home\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.323026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.321179 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-dshm\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.323026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.321304 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-model-cache\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.323026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.322758 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-home\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.323026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.323000 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-tmp-dir\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.323716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.323577 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-model-cache\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.328351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.326964 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d42774cd-c445-4c21-902e-6f832ec412c4-tls-certs\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.328351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.327317 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-dshm\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.353986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.353915 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrldf\" (UniqueName: \"kubernetes.io/projected/d42774cd-c445-4c21-902e-6f832ec412c4-kube-api-access-rrldf\") pod \"distinct-2-w4-fe1374-kserve-df849c688-n6jpw\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.425488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.425460 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:10.724616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.724595 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw"] Oct 30 18:14:10.726924 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:14:10.726887 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd42774cd_c445_4c21_902e_6f832ec412c4.slice/crio-df80348762f8b078031d6549487fb9cf77acfdd58351e3dcc21c72e1d4b9ede8 WatchSource:0}: Error finding container df80348762f8b078031d6549487fb9cf77acfdd58351e3dcc21c72e1d4b9ede8: Status 404 returned error can't find the container with id df80348762f8b078031d6549487fb9cf77acfdd58351e3dcc21c72e1d4b9ede8 Oct 30 18:14:10.811223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.811193 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" event={"ID":"d42774cd-c445-4c21-902e-6f832ec412c4","Type":"ContainerStarted","Data":"752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65"} Oct 30 18:14:10.811381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.811237 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" event={"ID":"d42774cd-c445-4c21-902e-6f832ec412c4","Type":"ContainerStarted","Data":"df80348762f8b078031d6549487fb9cf77acfdd58351e3dcc21c72e1d4b9ede8"} Oct 30 18:14:10.852504 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:10.852455 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podStartSLOduration=0.852441675 podStartE2EDuration="852.441675ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:10.850132266 +0000 UTC m=+949.899177179" watchObservedRunningTime="2026-10-30 18:14:10.852441675 +0000 UTC m=+949.901486588" Oct 30 18:14:11.211193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:11.211154 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:14:11.425586 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:11.425557 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:19.869720 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:14:19.869681 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:14:20.426685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:20.426659 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:20.439444 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:20.439424 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:20.891165 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:20.891138 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:14:26.506875 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.506841 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv"] Oct 30 18:14:26.513539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.513516 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.528539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.528513 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv"] Oct 30 18:14:26.537570 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.537548 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-45dc34\"/\"distinct-2-w2-45dc34-kserve-self-signed-certs\"" Oct 30 18:14:26.629656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.629617 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tmp-dir\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.629838 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.629724 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j2sx\" (UniqueName: \"kubernetes.io/projected/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-kube-api-access-8j2sx\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.629897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.629868 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tls-certs\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.629954 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.629929 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-home\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.630056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.630036 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-model-cache\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.630211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.630187 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-dshm\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.731611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.731574 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2sx\" (UniqueName: \"kubernetes.io/projected/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-kube-api-access-8j2sx\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.732472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.732406 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tls-certs\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.732654 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.732483 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-home\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.732654 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.732535 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-model-cache\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.732654 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.732632 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-dshm\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.732826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.732677 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tmp-dir\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.733211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.733187 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tmp-dir\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.734218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.733913 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-model-cache\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.734218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.734155 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-home\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.737032 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.736984 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tls-certs\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.739721 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.739700 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-dshm\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.752693 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.752674 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2sx\" (UniqueName: \"kubernetes.io/projected/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-kube-api-access-8j2sx\") pod \"distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:26.833111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:26.833085 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:27.132955 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:27.132932 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv"] Oct 30 18:14:27.135068 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:14:27.135037 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf92ff043_4e4c_4cb0_b03b_4d13a79a8a79.slice/crio-58044fe1c2a1a65c5fed2fe55679e289af4da8e86bebab27539fb4bef04850cd WatchSource:0}: Error finding container 58044fe1c2a1a65c5fed2fe55679e289af4da8e86bebab27539fb4bef04850cd: Status 404 returned error can't find the container with id 58044fe1c2a1a65c5fed2fe55679e289af4da8e86bebab27539fb4bef04850cd Oct 30 18:14:27.964632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:27.964593 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" event={"ID":"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79","Type":"ContainerStarted","Data":"9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d"} Oct 30 18:14:27.965062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:27.964643 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" event={"ID":"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79","Type":"ContainerStarted","Data":"58044fe1c2a1a65c5fed2fe55679e289af4da8e86bebab27539fb4bef04850cd"} Oct 30 18:14:28.004824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:28.004779 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podStartSLOduration=2.004767074 podStartE2EDuration="2.004767074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:28.003934414 +0000 UTC m=+967.052979340" watchObservedRunningTime="2026-10-30 18:14:28.004767074 +0000 UTC m=+967.053812021" Oct 30 18:14:28.833977 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:28.833943 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:36.075350 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:36.075316 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:14:36.833853 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:36.833821 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:36.854640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:36.854609 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:37.035600 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.035563 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:14:37.710289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.710174 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc"] Oct 30 18:14:37.715848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.715057 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.753284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.752372 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-380c08-openshift-default-dockercfg-74pzf\"" Oct 30 18:14:37.753284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.753128 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc"] Oct 30 18:14:37.875288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.869873 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47113181-eed4-4806-afa0-a05250b7b7a8-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.875288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.869944 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.875288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.869973 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47113181-eed4-4806-afa0-a05250b7b7a8-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.875288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.870011 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.875288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.870059 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.875288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.870117 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.875288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.870159 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.875288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.870211 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.875288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.870283 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlqjr\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-kube-api-access-mlqjr\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.977360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977143 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.977360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977211 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.977360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977252 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.977360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977327 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.977360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977363 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqjr\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-kube-api-access-mlqjr\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.977779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977417 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47113181-eed4-4806-afa0-a05250b7b7a8-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.977779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977473 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.977779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977499 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47113181-eed4-4806-afa0-a05250b7b7a8-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.977779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977539 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.978002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.977962 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.979291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.978674 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.979291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.979164 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:37.979503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:37.979393 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:38.004619 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.004558 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:38.004897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.004837 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:38.015368 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.008047 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqjr\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-kube-api-access-mlqjr\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:38.015368 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.008807 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47113181-eed4-4806-afa0-a05250b7b7a8-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:38.018995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.018974 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47113181-eed4-4806-afa0-a05250b7b7a8-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:38.046507 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.046486 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:38.450044 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.450010 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc"] Oct 30 18:14:38.460154 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.460123 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:38.460303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.460184 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:38.460303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:38.460232 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:39.034366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:39.034314 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" event={"ID":"47113181-eed4-4806-afa0-a05250b7b7a8","Type":"ContainerStarted","Data":"9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee"} Oct 30 18:14:39.034815 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:39.034385 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" event={"ID":"47113181-eed4-4806-afa0-a05250b7b7a8","Type":"ContainerStarted","Data":"37a88dcaf3a32bb4b25024894b40b40c74386e4a02a71b85feface4fc53d9fd8"} Oct 30 18:14:39.047668 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:39.047649 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:39.049232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:39.049207 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:14:39.049341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:39.049252 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 18:14:39.098451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:39.098399 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" podStartSLOduration=2.098381097 podStartE2EDuration="2.098381097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:39.087956368 +0000 UTC m=+978.137001278" watchObservedRunningTime="2026-10-30 18:14:39.098381097 +0000 UTC m=+978.147426010" Oct 30 18:14:40.050286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:40.049623 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:14:40.050286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:40.049678 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 18:14:41.051284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.050774 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:14:41.051284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.050834 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 18:14:41.853649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.853614 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg"] Oct 30 18:14:41.858520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.858498 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:41.894184 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.894157 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg"] Oct 30 18:14:41.963257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.963198 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:41.963492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.963300 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:41.963492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.963328 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2613d9-58e1-438d-b256-d6bc303cd461-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:41.963587 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.963489 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:41.963636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.963579 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:41.963705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.963675 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:41.963743 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.963709 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:41.963804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.963788 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gclrq\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-kube-api-access-gclrq\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:41.963842 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:41.963834 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.054401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.054366 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:14:42.054937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.054906 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 18:14:42.064909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.064880 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.064941 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.064975 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gclrq\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-kube-api-access-gclrq\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065117 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.065031 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065117 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.065111 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.065138 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.065163 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2613d9-58e1-438d-b256-d6bc303cd461-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.065215 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065444 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.065278 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065444 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.065367 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.065611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.065592 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.066060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.066005 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2613d9-58e1-438d-b256-d6bc303cd461-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.066060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.066011 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.066060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.066009 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.067748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.067728 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.071740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.071710 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.085033 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.085011 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gclrq\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-kube-api-access-gclrq\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.085129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.085018 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.174671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.174601 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:42.507116 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:14:42.507087 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2613d9_58e1_438d_b256_d6bc303cd461.slice/crio-8e087971f5ac8a0382b7553a4d48f949cfa7e6b5a178c3b5b25eeb4c980fc407 WatchSource:0}: Error finding container 8e087971f5ac8a0382b7553a4d48f949cfa7e6b5a178c3b5b25eeb4c980fc407: Status 404 returned error can't find the container with id 8e087971f5ac8a0382b7553a4d48f949cfa7e6b5a178c3b5b25eeb4c980fc407 Oct 30 18:14:42.507348 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.507099 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg"] Oct 30 18:14:42.509608 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.509577 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:42.509705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.509643 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:42.509705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:42.509678 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:43.050288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:43.049648 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.050288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:43.049708 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 18:14:43.078534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:43.078496 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" event={"ID":"2d2613d9-58e1-438d-b256-d6bc303cd461","Type":"ContainerStarted","Data":"b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc"} Oct 30 18:14:43.079098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:43.079072 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" event={"ID":"2d2613d9-58e1-438d-b256-d6bc303cd461","Type":"ContainerStarted","Data":"8e087971f5ac8a0382b7553a4d48f949cfa7e6b5a178c3b5b25eeb4c980fc407"} Oct 30 18:14:43.126079 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:43.126017 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podStartSLOduration=2.125997975 podStartE2EDuration="2.125997975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:43.123075483 +0000 UTC m=+982.172120400" watchObservedRunningTime="2026-10-30 18:14:43.125997975 +0000 UTC m=+982.175042888" Oct 30 18:14:43.179294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:43.179120 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:43.179294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:43.179252 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.179576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:43.179337 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:14:44.054681 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:44.054644 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.057070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:44.057038 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 18:14:44.178611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:44.178575 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.179214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:44.179183 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:14:45.047462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:45.047426 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.047747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:45.047716 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 18:14:45.175087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:45.175055 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.175288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:45.175107 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:14:46.175136 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.175095 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:14:46.175662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.175159 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:14:46.277976 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.277858 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:46.278708 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.278685 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:46.279374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.279355 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:14:46.760779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.760750 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r"] Oct 30 18:14:46.765899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.765864 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:46.772527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.772503 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r"] Oct 30 18:14:46.951011 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.950965 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:46.951319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.951295 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:46.951484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.951467 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:46.951609 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.951595 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:46.951732 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.951715 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:46.951851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.951838 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:46.951987 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.951972 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:46.952095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.952082 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrlq7\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-kube-api-access-jrlq7\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:46.952323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:46.952307 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054035 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054141 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054175 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054227 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054310 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054338 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054368 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054407 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054420 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrlq7\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-kube-api-access-jrlq7\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.054736 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.054670 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.055100 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.055077 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.055100 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.055093 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.055375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.055342 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.055534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.055514 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.056874 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.056841 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.057088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.057070 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.076893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.076863 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.076992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.076927 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrlq7\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-kube-api-access-jrlq7\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.084861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.084841 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:47.178200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.177763 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:14:47.178200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.177830 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:14:47.451288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.446618 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r"] Oct 30 18:14:47.459139 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:14:47.459106 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343b1bdb_a9bc_415c_bcf3_a3cf7de4200b.slice/crio-cd6955b1d027adc4ceeb6410c703b7293946fa1bb729d1d01ee2aee9c953c465 WatchSource:0}: Error finding container cd6955b1d027adc4ceeb6410c703b7293946fa1bb729d1d01ee2aee9c953c465: Status 404 returned error can't find the container with id cd6955b1d027adc4ceeb6410c703b7293946fa1bb729d1d01ee2aee9c953c465 Oct 30 18:14:47.461147 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.461116 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:47.461241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.461197 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:47.461336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:47.461242 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:14:48.121003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:48.118566 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" event={"ID":"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b","Type":"ContainerStarted","Data":"2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e"} Oct 30 18:14:48.121003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:48.118628 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" event={"ID":"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b","Type":"ContainerStarted","Data":"cd6955b1d027adc4ceeb6410c703b7293946fa1bb729d1d01ee2aee9c953c465"} Oct 30 18:14:48.178302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:48.175729 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:14:48.178302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:48.175803 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:14:49.085214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:49.085186 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:49.086909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:49.086884 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:49.087088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:49.086930 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:14:49.175364 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:49.175334 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 18:14:49.175501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:49.175399 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 18:14:49.892218 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:14:49.892180 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:14:50.085467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:50.085432 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:50.085634 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:50.085495 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:14:50.178780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:50.178701 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:50.179064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:50.179043 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:50.179600 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:50.179580 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:14:50.259288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:50.253448 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" podStartSLOduration=4.253427828 podStartE2EDuration="4.253427828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:48.190064538 +0000 UTC m=+987.239109448" watchObservedRunningTime="2026-10-30 18:14:50.253427828 +0000 UTC m=+989.302472742" Oct 30 18:14:51.085389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:51.085356 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:51.085774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:51.085405 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:14:52.085190 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:52.085157 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:52.085402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:52.085213 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:14:53.085413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:53.085388 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:53.085852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:53.085830 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:14:54.085789 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:54.085755 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:14:54.086159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:54.085833 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 18:14:55.089767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:55.089735 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:55.090139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:55.090050 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:14:55.090711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:14:55.090692 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:15:00.167717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.167634 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr"] Oct 30 18:15:00.173186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.173165 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:00.192601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.192564 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr"] Oct 30 18:15:00.201129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.201103 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:15:00.313403 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.313374 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053a90-9b53-4997-b57a-fd345bd76752-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr\" (UID: \"ea053a90-9b53-4997-b57a-fd345bd76752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:00.313557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.313412 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm2cr\" (UniqueName: \"kubernetes.io/projected/ea053a90-9b53-4997-b57a-fd345bd76752-kube-api-access-sm2cr\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr\" (UID: \"ea053a90-9b53-4997-b57a-fd345bd76752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:00.414660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.414626 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053a90-9b53-4997-b57a-fd345bd76752-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr\" (UID: \"ea053a90-9b53-4997-b57a-fd345bd76752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:00.414820 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.414665 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm2cr\" (UniqueName: \"kubernetes.io/projected/ea053a90-9b53-4997-b57a-fd345bd76752-kube-api-access-sm2cr\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr\" (UID: \"ea053a90-9b53-4997-b57a-fd345bd76752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:00.415294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.415246 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053a90-9b53-4997-b57a-fd345bd76752-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr\" (UID: \"ea053a90-9b53-4997-b57a-fd345bd76752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:00.434646 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.434584 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm2cr\" (UniqueName: \"kubernetes.io/projected/ea053a90-9b53-4997-b57a-fd345bd76752-kube-api-access-sm2cr\") pod \"maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr\" (UID: \"ea053a90-9b53-4997-b57a-fd345bd76752\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:00.486406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.486383 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:00.787087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.787057 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr"] Oct 30 18:15:00.788549 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:15:00.788525 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea053a90_9b53_4997_b57a_fd345bd76752.slice/crio-36dfbbae63986cf686654398540b58b6aecc906cc4a92baf07f479decffd49b5 WatchSource:0}: Error finding container 36dfbbae63986cf686654398540b58b6aecc906cc4a92baf07f479decffd49b5: Status 404 returned error can't find the container with id 36dfbbae63986cf686654398540b58b6aecc906cc4a92baf07f479decffd49b5 Oct 30 18:15:00.790840 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:00.790818 4862 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:15:01.285684 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:01.285651 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" event={"ID":"ea053a90-9b53-4997-b57a-fd345bd76752","Type":"ContainerStarted","Data":"36dfbbae63986cf686654398540b58b6aecc906cc4a92baf07f479decffd49b5"} Oct 30 18:15:04.306247 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:04.306215 4862 generic.go:356] "Generic (PLEG): container finished" podID="ea053a90-9b53-4997-b57a-fd345bd76752" containerID="6208faa217e14618c9a67ee5a61b7231956cb7036b08071ebdb27557bdeb319d" exitCode=60 Oct 30 18:15:04.306679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:04.306253 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" event={"ID":"ea053a90-9b53-4997-b57a-fd345bd76752","Type":"ContainerDied","Data":"6208faa217e14618c9a67ee5a61b7231956cb7036b08071ebdb27557bdeb319d"} Oct 30 18:15:04.306679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:04.306518 4862 scope.go:122] "RemoveContainer" containerID="6208faa217e14618c9a67ee5a61b7231956cb7036b08071ebdb27557bdeb319d" Oct 30 18:15:05.314645 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.314606 4862 generic.go:356] "Generic (PLEG): container finished" podID="ea053a90-9b53-4997-b57a-fd345bd76752" containerID="1a892bdba072bc765b9a9b1c0bf69186eee1901fb37dab23baf159238d700d7b" exitCode=60 Oct 30 18:15:05.315081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.314652 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" event={"ID":"ea053a90-9b53-4997-b57a-fd345bd76752","Type":"ContainerDied","Data":"1a892bdba072bc765b9a9b1c0bf69186eee1901fb37dab23baf159238d700d7b"} Oct 30 18:15:05.315081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.314711 4862 scope.go:122] "RemoveContainer" containerID="6208faa217e14618c9a67ee5a61b7231956cb7036b08071ebdb27557bdeb319d" Oct 30 18:15:05.315081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.314983 4862 scope.go:122] "RemoveContainer" containerID="1a892bdba072bc765b9a9b1c0bf69186eee1901fb37dab23baf159238d700d7b" Oct 30 18:15:05.315377 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:15:05.315353 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr_odh-ai-gateway-infra(ea053a90-9b53-4997-b57a-fd345bd76752)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" Oct 30 18:15:05.870020 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.869986 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz"] Oct 30 18:15:05.877432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.877394 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:05.892376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.892346 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz"] Oct 30 18:15:05.897541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.897517 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:15:05.993226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.993200 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wz6l\" (UniqueName: \"kubernetes.io/projected/c63b5e09-2f5e-4549-a13d-34632b46120d-kube-api-access-2wz6l\") pod \"maas-api-revoke-sub-4163c57f-d69bz\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:05.993406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.993334 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4163c57f-d69bz\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:05.993406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:05.993378 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4163c57f-d69bz\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:06.093988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:06.093964 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wz6l\" (UniqueName: \"kubernetes.io/projected/c63b5e09-2f5e-4549-a13d-34632b46120d-kube-api-access-2wz6l\") pod \"maas-api-revoke-sub-4163c57f-d69bz\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:06.094135 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:06.094034 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4163c57f-d69bz\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:06.094135 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:06.094083 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4163c57f-d69bz\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:06.094786 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:06.094765 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4163c57f-d69bz\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:06.096501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:06.096479 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4163c57f-d69bz\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:06.114178 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:06.114158 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wz6l\" (UniqueName: \"kubernetes.io/projected/c63b5e09-2f5e-4549-a13d-34632b46120d-kube-api-access-2wz6l\") pod \"maas-api-revoke-sub-4163c57f-d69bz\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:06.191957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:06.191887 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:06.328653 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:06.327183 4862 scope.go:122] "RemoveContainer" containerID="1a892bdba072bc765b9a9b1c0bf69186eee1901fb37dab23baf159238d700d7b" Oct 30 18:15:06.328653 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:15:06.327490 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr_odh-ai-gateway-infra(ea053a90-9b53-4997-b57a-fd345bd76752)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" Oct 30 18:15:06.497053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:06.497027 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz"] Oct 30 18:15:06.498725 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:15:06.498697 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63b5e09_2f5e_4549_a13d_34632b46120d.slice/crio-cbb27d9efc4903e9b207ed7dce490f809aac79157055b7e29b5691a40c36020d WatchSource:0}: Error finding container cbb27d9efc4903e9b207ed7dce490f809aac79157055b7e29b5691a40c36020d: Status 404 returned error can't find the container with id cbb27d9efc4903e9b207ed7dce490f809aac79157055b7e29b5691a40c36020d Oct 30 18:15:07.334225 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:07.334190 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" event={"ID":"c63b5e09-2f5e-4549-a13d-34632b46120d","Type":"ContainerStarted","Data":"33bd4f40f1a112d1145da39b82d7a32f8fdf60d4906159942e96ca70eaa191b0"} Oct 30 18:15:07.334704 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:07.334238 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" event={"ID":"c63b5e09-2f5e-4549-a13d-34632b46120d","Type":"ContainerStarted","Data":"cbb27d9efc4903e9b207ed7dce490f809aac79157055b7e29b5691a40c36020d"} Oct 30 18:15:07.376326 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:07.376278 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" podStartSLOduration=2.376254146 podStartE2EDuration="2.376254146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:07.374476001 +0000 UTC m=+1006.423520924" watchObservedRunningTime="2026-10-30 18:15:07.376254146 +0000 UTC m=+1006.425299058" Oct 30 18:15:08.341226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:08.341192 4862 generic.go:356] "Generic (PLEG): container finished" podID="c63b5e09-2f5e-4549-a13d-34632b46120d" containerID="33bd4f40f1a112d1145da39b82d7a32f8fdf60d4906159942e96ca70eaa191b0" exitCode=0 Oct 30 18:15:08.341653 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:08.341310 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" event={"ID":"c63b5e09-2f5e-4549-a13d-34632b46120d","Type":"ContainerDied","Data":"33bd4f40f1a112d1145da39b82d7a32f8fdf60d4906159942e96ca70eaa191b0"} Oct 30 18:15:09.489903 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.489878 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:09.631440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.631364 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63b5e09-2f5e-4549-a13d-34632b46120d-kube-api-access-2wz6l\" (UniqueName: \"kubernetes.io/projected/c63b5e09-2f5e-4549-a13d-34632b46120d-kube-api-access-2wz6l\") pod \"c63b5e09-2f5e-4549-a13d-34632b46120d\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " Oct 30 18:15:09.631440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.631408 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-xks-ca\") pod \"c63b5e09-2f5e-4549-a13d-34632b46120d\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " Oct 30 18:15:09.631664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.631487 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-openshift-ca\") pod \"c63b5e09-2f5e-4549-a13d-34632b46120d\" (UID: \"c63b5e09-2f5e-4549-a13d-34632b46120d\") " Oct 30 18:15:09.631903 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.631876 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-openshift-ca" pod "c63b5e09-2f5e-4549-a13d-34632b46120d" (UID: "c63b5e09-2f5e-4549-a13d-34632b46120d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:09.633689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.633662 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63b5e09-2f5e-4549-a13d-34632b46120d-kube-api-access-2wz6l" pod "c63b5e09-2f5e-4549-a13d-34632b46120d" (UID: "c63b5e09-2f5e-4549-a13d-34632b46120d"). InnerVolumeSpecName "kube-api-access-2wz6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:09.633791 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.633750 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-xks-ca" pod "c63b5e09-2f5e-4549-a13d-34632b46120d" (UID: "c63b5e09-2f5e-4549-a13d-34632b46120d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:09.733070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.733049 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wz6l\" (UniqueName: \"kubernetes.io/projected/c63b5e09-2f5e-4549-a13d-34632b46120d-kube-api-access-2wz6l\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:15:09.733070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.733072 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:15:09.733215 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:09.733081 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c63b5e09-2f5e-4549-a13d-34632b46120d-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:15:10.353909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:10.353878 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" Oct 30 18:15:10.353909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:10.353894 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz" event={"ID":"c63b5e09-2f5e-4549-a13d-34632b46120d","Type":"ContainerDied","Data":"cbb27d9efc4903e9b207ed7dce490f809aac79157055b7e29b5691a40c36020d"} Oct 30 18:15:10.354106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:10.353944 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbb27d9efc4903e9b207ed7dce490f809aac79157055b7e29b5691a40c36020d" Oct 30 18:15:15.051818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:15.051785 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:15:17.501415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.501370 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.514387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.514343 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.564654 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.564606 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.614401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.614352 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.637939 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.637897 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.708339 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.708290 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.861286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.860350 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.885287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.861594 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:17.949920 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.949781 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:17.949920 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:17.949848 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:18.061379 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.061348 4862 scope.go:122] "RemoveContainer" containerID="1a892bdba072bc765b9a9b1c0bf69186eee1901fb37dab23baf159238d700d7b" Oct 30 18:15:18.140332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.133410 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:18.140332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.133479 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:18.157559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.157523 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:18.412450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.412368 4862 patch_prober.go:29] interesting pod/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:18.412696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.412666 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:18.432285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.429370 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:18.641401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.641358 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 18:15:18.643334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.642247 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:15:18.711483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.711386 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 18:15:18.712242 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.712219 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:15:18.922952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:18.922926 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:15:19.025186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:19.025101 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:15:19.494075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:19.494036 4862 generic.go:356] "Generic (PLEG): container finished" podID="ea053a90-9b53-4997-b57a-fd345bd76752" containerID="978d12e502e3017ccd930d029b2869ab9eb7450048ee637affcaa99812b1049c" exitCode=60 Oct 30 18:15:19.494075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:19.494063 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" event={"ID":"ea053a90-9b53-4997-b57a-fd345bd76752","Type":"ContainerDied","Data":"978d12e502e3017ccd930d029b2869ab9eb7450048ee637affcaa99812b1049c"} Oct 30 18:15:19.494422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:19.494131 4862 scope.go:122] "RemoveContainer" containerID="1a892bdba072bc765b9a9b1c0bf69186eee1901fb37dab23baf159238d700d7b" Oct 30 18:15:19.494964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:19.494933 4862 scope.go:122] "RemoveContainer" containerID="978d12e502e3017ccd930d029b2869ab9eb7450048ee637affcaa99812b1049c" Oct 30 18:15:19.495428 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:15:19.495399 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr_odh-ai-gateway-infra(ea053a90-9b53-4997-b57a-fd345bd76752)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" Oct 30 18:15:19.872091 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:15:19.872053 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:15:20.551046 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:20.551014 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr"] Oct 30 18:15:20.740983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:20.740959 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:20.826350 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:20.825674 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea053a90-9b53-4997-b57a-fd345bd76752-kube-api-access-sm2cr\" (UniqueName: \"kubernetes.io/projected/ea053a90-9b53-4997-b57a-fd345bd76752-kube-api-access-sm2cr\") pod \"ea053a90-9b53-4997-b57a-fd345bd76752\" (UID: \"ea053a90-9b53-4997-b57a-fd345bd76752\") " Oct 30 18:15:20.826350 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:20.825743 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea053a90-9b53-4997-b57a-fd345bd76752-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053a90-9b53-4997-b57a-fd345bd76752-maas-api-service-ca\") pod \"ea053a90-9b53-4997-b57a-fd345bd76752\" (UID: \"ea053a90-9b53-4997-b57a-fd345bd76752\") " Oct 30 18:15:20.827049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:20.827010 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea053a90-9b53-4997-b57a-fd345bd76752-maas-api-service-ca" pod "ea053a90-9b53-4997-b57a-fd345bd76752" (UID: "ea053a90-9b53-4997-b57a-fd345bd76752"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:20.828459 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:20.828436 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea053a90-9b53-4997-b57a-fd345bd76752-kube-api-access-sm2cr" pod "ea053a90-9b53-4997-b57a-fd345bd76752" (UID: "ea053a90-9b53-4997-b57a-fd345bd76752"). InnerVolumeSpecName "kube-api-access-sm2cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:20.927105 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:20.927072 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm2cr\" (UniqueName: \"kubernetes.io/projected/ea053a90-9b53-4997-b57a-fd345bd76752-kube-api-access-sm2cr\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:15:20.927105 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:20.927106 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053a90-9b53-4997-b57a-fd345bd76752-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.511373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:21.511342 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" Oct 30 18:15:21.511553 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:21.511345 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr" event={"ID":"ea053a90-9b53-4997-b57a-fd345bd76752","Type":"ContainerDied","Data":"36dfbbae63986cf686654398540b58b6aecc906cc4a92baf07f479decffd49b5"} Oct 30 18:15:21.511553 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:21.511484 4862 scope.go:122] "RemoveContainer" containerID="978d12e502e3017ccd930d029b2869ab9eb7450048ee637affcaa99812b1049c" Oct 30 18:15:21.562870 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:21.562847 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr"] Oct 30 18:15:21.581983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:21.576478 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-de7e3a-29889735-7g4zr"] Oct 30 18:15:22.832854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.832826 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6"] Oct 30 18:15:22.833534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833515 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c63b5e09-2f5e-4549-a13d-34632b46120d" containerName="revoke-keys" Oct 30 18:15:22.833643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833535 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63b5e09-2f5e-4549-a13d-34632b46120d" containerName="revoke-keys" Oct 30 18:15:22.833643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833558 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" containerName="cleanup" Oct 30 18:15:22.833643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833565 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" containerName="cleanup" Oct 30 18:15:22.833643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833597 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" containerName="cleanup" Oct 30 18:15:22.833643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833604 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" containerName="cleanup" Oct 30 18:15:22.833643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833615 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" containerName="cleanup" Oct 30 18:15:22.833643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833622 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" containerName="cleanup" Oct 30 18:15:22.833975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833750 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" containerName="cleanup" Oct 30 18:15:22.833975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833762 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" containerName="cleanup" Oct 30 18:15:22.833975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.833779 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63b5e09-2f5e-4549-a13d-34632b46120d" containerName="revoke-keys" Oct 30 18:15:22.838480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.838462 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:22.857955 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.857931 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6"] Oct 30 18:15:22.872377 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.872355 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc"] Oct 30 18:15:22.873055 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.873031 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" containerName="cleanup" Oct 30 18:15:22.876370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.876354 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.891394 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.891373 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc"] Oct 30 18:15:22.947268 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947242 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.947378 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947307 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrvz\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-kube-api-access-4jrvz\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.947419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947387 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8fzt\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-kube-api-access-z8fzt\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:22.947454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947443 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.947538 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947521 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.947581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947569 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:22.947615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947603 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:22.947654 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947632 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:22.947699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947669 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.947699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947685 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b5882-396e-446e-9efc-b7c11e0d454a-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:22.947773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947761 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:22.947805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947782 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.947837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947826 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:22.947924 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947913 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3b36b7-bcaa-40be-a051-8694cd685308-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.947961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947935 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.947993 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.947970 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:22.948035 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.948020 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:22.948072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:22.948054 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.049408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049387 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.049544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049416 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.049544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049434 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.049544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049466 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.049544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049490 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrvz\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-kube-api-access-4jrvz\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.049544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049527 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8fzt\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-kube-api-access-z8fzt\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.049835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049577 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.049835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049617 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.049835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049654 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.049835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049698 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.049835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049722 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.049835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049749 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.049835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049774 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b5882-396e-446e-9efc-b7c11e0d454a-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.049835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049830 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.050370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049836 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.050370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049905 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.050370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049947 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.050370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049953 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.050370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.049985 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3b36b7-bcaa-40be-a051-8694cd685308-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.050370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.050012 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.050370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.050185 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.050370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.050211 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.050837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.050650 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.050927 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.050905 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3b36b7-bcaa-40be-a051-8694cd685308-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.051025 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.051002 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.051200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.051174 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.051313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.051203 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.051402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.051311 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b5882-396e-446e-9efc-b7c11e0d454a-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.052408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.052386 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.052553 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.052532 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.052655 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.052639 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.052791 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.052771 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.063163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.063136 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea053a90-9b53-4997-b57a-fd345bd76752" path="/var/lib/kubelet/pods/ea053a90-9b53-4997-b57a-fd345bd76752/volumes" Oct 30 18:15:23.069035 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.069012 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.069226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.069202 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrvz\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-kube-api-access-4jrvz\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.069965 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.069943 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.069965 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.069962 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8fzt\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-kube-api-access-z8fzt\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.151388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.151313 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:23.203788 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.203753 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:23.476781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.476698 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6"] Oct 30 18:15:23.481009 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:15:23.480983 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8b5882_396e_446e_9efc_b7c11e0d454a.slice/crio-84425179de4f93472b8d0418b301fce3cc564d49f0b54fcaf577342725e9176f WatchSource:0}: Error finding container 84425179de4f93472b8d0418b301fce3cc564d49f0b54fcaf577342725e9176f: Status 404 returned error can't find the container with id 84425179de4f93472b8d0418b301fce3cc564d49f0b54fcaf577342725e9176f Oct 30 18:15:23.483627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.483596 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:23.483701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.483661 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:23.483734 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.483698 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:23.525153 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.525125 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" event={"ID":"ef8b5882-396e-446e-9efc-b7c11e0d454a","Type":"ContainerStarted","Data":"84425179de4f93472b8d0418b301fce3cc564d49f0b54fcaf577342725e9176f"} Oct 30 18:15:23.527195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.527171 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc"] Oct 30 18:15:23.528679 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:15:23.528659 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3b36b7_bcaa_40be_a051_8694cd685308.slice/crio-584adebf2a13a2d2330bd1f10be3a05216611fff9dc25a60d6704079bd2b4eb5 WatchSource:0}: Error finding container 584adebf2a13a2d2330bd1f10be3a05216611fff9dc25a60d6704079bd2b4eb5: Status 404 returned error can't find the container with id 584adebf2a13a2d2330bd1f10be3a05216611fff9dc25a60d6704079bd2b4eb5 Oct 30 18:15:23.530999 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.530973 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:23.531067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.531031 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:23.531067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:23.531061 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:24.581893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:24.581841 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" event={"ID":"5a3b36b7-bcaa-40be-a051-8694cd685308","Type":"ContainerStarted","Data":"8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b"} Oct 30 18:15:24.582582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:24.582557 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" event={"ID":"5a3b36b7-bcaa-40be-a051-8694cd685308","Type":"ContainerStarted","Data":"584adebf2a13a2d2330bd1f10be3a05216611fff9dc25a60d6704079bd2b4eb5"} Oct 30 18:15:24.584809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:24.584788 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" event={"ID":"ef8b5882-396e-446e-9efc-b7c11e0d454a","Type":"ContainerStarted","Data":"12f59539367b860ab9c9ebadae64280523069b75d7d8d8079a90ba5f4b849339"} Oct 30 18:15:24.669385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:24.668843 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" podStartSLOduration=2.668819738 podStartE2EDuration="2.668819738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:24.660597385 +0000 UTC m=+1023.709642302" watchObservedRunningTime="2026-10-30 18:15:24.668819738 +0000 UTC m=+1023.717864652" Oct 30 18:15:24.767286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:24.766599 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" podStartSLOduration=2.766577748 podStartE2EDuration="2.766577748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:24.726120748 +0000 UTC m=+1023.775165660" watchObservedRunningTime="2026-10-30 18:15:24.766577748 +0000 UTC m=+1023.815622661" Oct 30 18:15:25.154714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:25.153731 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:25.158284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:25.156580 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:15:25.158284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:25.156644 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:15:25.204342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:25.204312 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:25.205765 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:25.205744 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:15:25.205838 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:25.205791 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:15:26.154036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:26.153605 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:15:26.154036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:26.153663 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:15:26.205024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:26.204995 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:15:26.205189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:26.205040 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:15:26.983934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:26.983892 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt"] Oct 30 18:15:26.988492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:26.988472 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.015198 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.015174 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt"] Oct 30 18:15:27.018805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.018681 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.018941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.018851 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.018941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.018881 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2nrv\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-kube-api-access-d2nrv\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.019048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.018939 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.019048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.018979 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.019048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.019000 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.019226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.019070 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.019226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.019115 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45344089-c81d-4ee0-8615-b6ad76d4027d-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.019226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.019175 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.120237 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.120206 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.120454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.120242 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2nrv\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-kube-api-access-d2nrv\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.120454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.120314 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.120454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.120350 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.120454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.120370 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.120454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.120431 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.120725 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.120484 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45344089-c81d-4ee0-8615-b6ad76d4027d-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.120725 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.120542 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.120725 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.120616 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.121426 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.121396 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-credential-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.121573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.121454 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-socket\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.121712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.121690 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-data\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.121933 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.121907 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-certs\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.122060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.122037 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45344089-c81d-4ee0-8615-b6ad76d4027d-istiod-ca-cert\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.123359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.123340 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-podinfo\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.124515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.124495 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-envoy\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.153285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.152301 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:15:27.153285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.152368 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:15:27.177292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.176967 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-token\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.177292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.177049 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2nrv\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-kube-api-access-d2nrv\") pod \"e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.204669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.204640 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:15:27.204796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.204694 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:15:27.304359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.304328 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:27.682082 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.682057 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt"] Oct 30 18:15:27.687193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.687159 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:27.687315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.687240 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:27.687376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:27.687320 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:28.154287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:28.153700 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.154287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:28.153763 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:15:28.204979 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:28.204946 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.205558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:28.205525 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:15:28.619123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:28.619084 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" event={"ID":"45344089-c81d-4ee0-8615-b6ad76d4027d","Type":"ContainerStarted","Data":"8732922daf4c4d1bae6bff36377f44fcf6e4b912eb364d0509778dc63a3eca10"} Oct 30 18:15:28.619315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:28.619164 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" event={"ID":"45344089-c81d-4ee0-8615-b6ad76d4027d","Type":"ContainerStarted","Data":"6452b078b9444b27b8988925f374040302a2840780a207b1c0ee4bcedc21303c"} Oct 30 18:15:29.177295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:29.153620 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.177295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:29.153678 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:15:29.205750 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:29.205718 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.207157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:29.206480 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:15:29.305410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:29.305327 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:29.307353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:29.307320 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.307512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:29.307376 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:15:30.160755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:30.160701 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.160963 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:30.160772 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 18:15:30.204801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:30.204771 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.205026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:30.204818 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:15:30.305399 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:30.305366 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.305769 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:30.305427 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:15:31.224528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.224443 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:31.224891 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.224870 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:31.227805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.227404 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:15:31.241294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.240818 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:31.242182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.242161 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:31.258307 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.257436 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 18:15:31.284074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.284028 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podStartSLOduration=5.284012287 podStartE2EDuration="5.284012287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:28.673626536 +0000 UTC m=+1027.722671450" watchObservedRunningTime="2026-10-30 18:15:31.284012287 +0000 UTC m=+1030.333057201" Oct 30 18:15:31.304965 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.304925 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:15:31.305119 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.305008 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:15:31.966289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.961532 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p"] Oct 30 18:15:31.988457 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.988422 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p"] Oct 30 18:15:31.988632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:31.988536 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.020710 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.020675 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.020935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.020915 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.021096 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.021077 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhf2q\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-kube-api-access-xhf2q\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.021221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.021207 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.021397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.021381 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.021514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.021499 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.021628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.021613 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.021751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.021737 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.021907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.021890 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.123087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.123046 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.129513 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.129478 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.129718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.129702 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.133449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.133414 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhf2q\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-kube-api-access-xhf2q\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.156425 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.156391 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.156740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.156719 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.156898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.156881 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.157076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.157061 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.157254 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.157239 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.157836 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.157813 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.157974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.135886 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.163527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.158338 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.163527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.160097 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.163527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.136509 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.174771 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.168616 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.180500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.180449 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.183637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.183566 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.186906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.186880 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhf2q\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-kube-api-access-xhf2q\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:32.318320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.309050 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:15:32.318320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.309110 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:15:32.318320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:32.309729 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:33.346287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.320500 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:15:33.346287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.320556 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:15:33.400287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.394446 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p"] Oct 30 18:15:33.412781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.408525 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.435387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.435349 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.535687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.534368 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.537030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.536011 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:15:33.568286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.565329 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:33.568286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.565423 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:33.568286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.565467 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:33.617638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.596411 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.639689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.619302 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.761321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.757829 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" event={"ID":"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa","Type":"ContainerStarted","Data":"4495e3f0659b8b99552d6790795296e1e0e059d830bba5fef7266d7da5a25285"} Oct 30 18:15:33.798923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.796867 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.798923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.797170 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.840788 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.840744 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:33.934231 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:33.934182 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 18:15:34.035251 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.033681 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.036341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.036308 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.331287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.330730 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:15:34.331287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.330787 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:15:34.446287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.438394 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 18:15:34.446287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.439161 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:15:34.542290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.534716 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.544237 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.542670 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.544864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.544460 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:15:34.561331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.545101 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:15:34.597767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.597671 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.666796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.623278 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.667604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.667586 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:15:34.668292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.668276 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:15:34.823409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.823360 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:34.824028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.824004 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:15:34.846461 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.846425 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:34.870347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.867397 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:15:34.936303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.932182 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" event={"ID":"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa","Type":"ContainerStarted","Data":"923cf4d0528d3866e750dee6e85d9f4c15679b29acd2e45a454b214d91d9af24"} Oct 30 18:15:34.962125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.962086 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 18:15:34.972279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:34.972235 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:15:35.046581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.046538 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.047450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.047427 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:15:35.185005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.169337 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.413314 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.411642 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.479317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.479192 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 18:15:35.516812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.516764 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:35.524287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.520342 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 18:15:35.558202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.558163 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 30 18:15:35.558408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.558318 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:15:35.587736 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.587702 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 18:15:35.694135 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.693646 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.705341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.705300 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:15:35.705630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.705585 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:15:35.816289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.815923 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.871290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.869720 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:35.924289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.899362 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:15:36.020520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.020493 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:36.020806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.007276 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.021353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:35.958566 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.041900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.041867 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.050427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.050379 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 18:15:36.133506 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.133406 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.176405 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.176359 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.177042 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.177019 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:15:36.241462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.241431 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:36.241697 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.241669 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.353483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.353450 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:36.353740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.353714 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 18:15:36.353896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.353883 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:15:36.353998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.353984 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:15:36.536343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.463320 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.654220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.598345 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.655423 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.599077 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:36.664285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.611650 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podStartSLOduration=5.611626139 podStartE2EDuration="5.611626139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:34.99478413 +0000 UTC m=+1034.043829059" watchObservedRunningTime="2026-10-30 18:15:36.611626139 +0000 UTC m=+1035.660671053" Oct 30 18:15:36.681880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.654910 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:15:36.729927 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.729885 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 18:15:36.739409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.730188 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:15:36.824317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.816382 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:15:36.916108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:36.916066 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:37.027408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.027361 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.028012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.027982 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 18:15:37.042385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.042351 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.042915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.042895 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:15:37.069293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.067344 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.180757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.180333 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 18:15:37.205289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.196475 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.354847 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.354812 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:37.355155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.355109 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 18:15:37.358358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.358332 4862 patch_prober.go:29] interesting pod/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:15:37.358563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.358540 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:15:37.418699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.418662 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:15:37.654108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.654030 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.659380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.655242 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 18:15:37.667340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.665874 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 18:15:37.743034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.742109 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.748287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.743637 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:15:37.852310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.836491 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:15:37.852310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.842331 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.864041 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.862734 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:15:37.870828 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.870359 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:37.917220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:37.917127 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:38.053353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.052716 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:38.053526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.053408 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:15:38.080091 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.078385 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:38.186464 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.186363 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:38.291357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.287666 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:15:38.291781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.291751 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:15:38.313768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.313605 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:15:38.322140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.322117 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:38.322371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.322344 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 18:15:38.342006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.341982 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:15:38.342537 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.342518 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:15:38.357432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.357412 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:15:38.363806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.363788 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:15:38.367477 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.367458 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:15:38.387299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.386672 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:15:38.389898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.388857 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:15:38.642036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.638627 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:38.648376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.648347 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:38.667316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:38.666560 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:15:39.311106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:39.311076 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:39.311300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:39.311125 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 18:15:40.310436 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:40.310401 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:40.310937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:40.310456 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 18:15:41.310723 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:41.310689 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:41.311209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:41.310741 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 18:15:42.310906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:42.310870 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:15:42.311334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:42.310940 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 18:15:43.409995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:43.409965 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:43.410851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:43.410796 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:43.412188 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:43.412165 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:15:49.871051 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:15:49.871017 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:15:54.024617 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.024583 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j"] Oct 30 18:15:54.028295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.028243 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.064283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.061902 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j"] Oct 30 18:15:54.125362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.125334 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.125521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.125410 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.125521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.125480 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.125612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.125589 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.125612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.125608 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.125917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.125883 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.126012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.125940 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.126078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.126060 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.126132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.126087 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr45m\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-kube-api-access-nr45m\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.227539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227506 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.227719 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227554 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.227719 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227606 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.227719 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227621 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr45m\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-kube-api-access-nr45m\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.227876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227745 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.227876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227848 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.227986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227880 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.227986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227942 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.227986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227966 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.228136 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.228062 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.228136 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.227963 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.228226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.228198 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.228288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.228233 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.228653 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.228582 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.230371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.230348 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.230588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.230568 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.253118 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.253083 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr45m\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-kube-api-access-nr45m\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.260648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.260618 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.343212 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.343178 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:54.371361 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.371334 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:15:54.661695 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.661620 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j"] Oct 30 18:15:54.665921 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:15:54.665892 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd3d2ab_2f20_480c_8d11_32012e08b53d.slice/crio-f68c6f49232c79f1319988d30607f0417e91d3a48ce0550467ef4f147376330e WatchSource:0}: Error finding container f68c6f49232c79f1319988d30607f0417e91d3a48ce0550467ef4f147376330e: Status 404 returned error can't find the container with id f68c6f49232c79f1319988d30607f0417e91d3a48ce0550467ef4f147376330e Oct 30 18:15:54.668749 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.668708 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:54.668843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.668795 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:54.668843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:54.668834 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:15:55.680323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:55.679235 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" event={"ID":"ebd3d2ab-2f20-480c-8d11-32012e08b53d","Type":"ContainerStarted","Data":"63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0"} Oct 30 18:15:55.680323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:55.679383 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" event={"ID":"ebd3d2ab-2f20-480c-8d11-32012e08b53d","Type":"ContainerStarted","Data":"f68c6f49232c79f1319988d30607f0417e91d3a48ce0550467ef4f147376330e"} Oct 30 18:15:55.814294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:55.813046 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podStartSLOduration=1.8130283820000002 podStartE2EDuration="1.813028382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:55.807836691 +0000 UTC m=+1054.856881605" watchObservedRunningTime="2026-10-30 18:15:55.813028382 +0000 UTC m=+1054.862073296" Oct 30 18:15:56.343723 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:56.343646 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:15:56.347196 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:56.346776 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:15:56.347196 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:56.346849 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 18:15:56.616338 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:56.615361 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:56.902526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:56.902418 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:56.903496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:56.903467 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:57.026529 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.026486 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.070536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.070490 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 30 18:15:57.071190 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.071154 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:15:57.188744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.188354 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded" Oct 30 18:15:57.293548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.292363 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 18:15:57.293848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.293815 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.347696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.347657 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:15:57.347969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.347940 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 18:15:57.370390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.370354 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:57.370587 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.370414 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.375392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.375342 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:15:57.402392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.402351 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.484489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.484383 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.618320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.616340 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 18:15:57.618320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.617137 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:15:57.862376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.854478 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.864871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.864743 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 18:15:57.916690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.908832 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:57.916690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.909172 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 18:15:57.916690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.909801 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:15:57.916690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:57.909858 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:15:58.033598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.033560 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:58.041905 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.041881 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:15:58.120694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.120546 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:15:58.215914 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.201162 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:58.219201 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.217176 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:15:58.305781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.305735 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 18:15:58.307348 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.307312 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:58.308992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.308971 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:15:58.309204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.309190 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:15:58.423834 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.384968 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:15:58.423834 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.386227 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:15:58.444120 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.444090 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:15:58.463561 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.463523 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 18:15:58.481724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.481697 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:15:58.483655 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.483631 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:15:58.483852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.483828 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 18:15:58.495414 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.495379 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:58.495991 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.495961 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 18:15:58.497415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.497398 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:15:58.931285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.923587 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 18:15:58.931285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:58.924957 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 18:15:59.057374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.057331 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 18:15:59.100745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.066515 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.100745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.071948 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:15:59.245024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.244928 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:15:59.321363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.321319 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.321935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.321600 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.440297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.439374 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:15:59.440297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.439432 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 18:15:59.461303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.445346 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.461303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.445679 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Oct 30 18:15:59.500340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.468950 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.500340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.498670 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded" Oct 30 18:15:59.578296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.573970 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.688010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.680089 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:15:59.772344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.753138 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.912314 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.901415 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-w8sfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:59.912314 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.901479 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.953315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.924243 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:59.953315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:15:59.925408 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 18:16:00.111298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.111061 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.223790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.222282 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:16:00.223790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.223531 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.247018 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.246962 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.250329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.247336 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 18:16:00.266559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.266532 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wgrr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:00.268349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.268321 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.334062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.334027 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 18:16:00.348776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.336328 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.349209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.343556 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:16:00.349521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.349493 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 18:16:00.404732 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.404655 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:16:00.412661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.412638 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:16:00.492042 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.489508 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:00.493104 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.493078 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:16:00.508553 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.508451 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:16:00.521312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.521293 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:16:00.590285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.582833 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:16:00.650155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.650119 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:16:00.690901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.690827 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:16:00.707051 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.705521 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:16:00.711071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:00.711048 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:16:01.353661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:01.353619 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:16:01.354163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:01.353684 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 18:16:02.354739 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:02.354702 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:16:02.355239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:02.354764 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 18:16:03.344221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:03.344189 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:16:03.344413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:03.344243 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 18:16:04.666176 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:04.666138 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:16:04.666722 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:04.666486 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:16:04.667507 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:04.667487 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 18:16:06.345358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.345325 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v"] Oct 30 18:16:06.348391 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.348372 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.372090 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.372059 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v"] Oct 30 18:16:06.493065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.493033 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.493248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.493084 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.493248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.493228 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzzr\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-kube-api-access-nvzzr\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.493404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.493344 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.493404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.493366 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.493404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.493388 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.493560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.493492 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.493560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.493517 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.493560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.493536 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595055 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595029 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzzr\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-kube-api-access-nvzzr\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595077 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595097 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595139 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595182 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595206 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595237 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595389 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595452 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595601 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595650 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595771 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595698 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.595837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595778 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.596057 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.595998 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.597896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.597872 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.598003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.597943 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.614999 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.614979 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzzr\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-kube-api-access-nvzzr\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.615094 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.615018 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-xhh6v\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:06.665014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:06.664994 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:07.025434 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:16:07.025401 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ae0d79_41b8_4c59_bcb4_225820e3b21d.slice/crio-335b39e8fa62a6be60add43332faf03b1a955266bac4dad8466e93524d64c0b0 WatchSource:0}: Error finding container 335b39e8fa62a6be60add43332faf03b1a955266bac4dad8466e93524d64c0b0: Status 404 returned error can't find the container with id 335b39e8fa62a6be60add43332faf03b1a955266bac4dad8466e93524d64c0b0 Oct 30 18:16:07.026295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:07.026273 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v"] Oct 30 18:16:07.027906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:07.027874 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:07.027964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:07.027938 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:07.028004 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:07.027966 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:07.303078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:07.303044 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" event={"ID":"09ae0d79-41b8-4c59-bcb4-225820e3b21d","Type":"ContainerStarted","Data":"b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0"} Oct 30 18:16:07.303253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:07.303105 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" event={"ID":"09ae0d79-41b8-4c59-bcb4-225820e3b21d","Type":"ContainerStarted","Data":"335b39e8fa62a6be60add43332faf03b1a955266bac4dad8466e93524d64c0b0"} Oct 30 18:16:07.345639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:07.345579 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" podStartSLOduration=1.345560382 podStartE2EDuration="1.345560382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:07.344341158 +0000 UTC m=+1066.393386075" watchObservedRunningTime="2026-10-30 18:16:07.345560382 +0000 UTC m=+1066.394605296" Oct 30 18:16:07.665141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:07.665053 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:08.671904 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:08.671865 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xhh6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:16:08.672370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:08.671923 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:16:09.100199 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.100159 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn"] Oct 30 18:16:09.102903 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.102881 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.154758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.154719 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.155059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.155041 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.155209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.155194 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.155444 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.155426 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.155849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.155832 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.156034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.156020 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa04282-d984-44f3-a442-655532f052af-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.156182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.156169 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.156434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.156405 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa04282-d984-44f3-a442-655532f052af-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.156628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.156612 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4bc4\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-kube-api-access-q4bc4\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.156830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.156333 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn"] Oct 30 18:16:09.258018 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.257983 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.258223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.258036 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.258223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.258065 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.258223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.258168 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.258223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.258201 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.258491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.258293 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa04282-d984-44f3-a442-655532f052af-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.258491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.258328 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.258491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.258365 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa04282-d984-44f3-a442-655532f052af-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.258876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.258837 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.259031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.259010 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.259123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.259109 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.259686 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.259656 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4bc4\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-kube-api-access-q4bc4\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.259843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.259820 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.260295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.260246 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa04282-d984-44f3-a442-655532f052af-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.262139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.262110 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.271576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.271553 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa04282-d984-44f3-a442-655532f052af-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.278019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.277998 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.279883 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.279866 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4bc4\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-kube-api-access-q4bc4\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.419328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.419227 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:09.665451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.665410 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xhh6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:16:09.665614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.665479 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:16:09.767188 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.767160 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn"] Oct 30 18:16:09.768208 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:16:09.768183 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa04282_d984_44f3_a442_655532f052af.slice/crio-d89bffed60190a9ab95494b5eb00715d7feeba6d2461e4ea434ff8272e3f7f81 WatchSource:0}: Error finding container d89bffed60190a9ab95494b5eb00715d7feeba6d2461e4ea434ff8272e3f7f81: Status 404 returned error can't find the container with id d89bffed60190a9ab95494b5eb00715d7feeba6d2461e4ea434ff8272e3f7f81 Oct 30 18:16:09.770629 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.770600 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.770702 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.770692 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:09.770735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:09.770720 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:10.323371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:10.323339 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" event={"ID":"bfa04282-d984-44f3-a442-655532f052af","Type":"ContainerStarted","Data":"f4e1655fa8eec133bfef33fdd10a62fe7d00c083a139bb6bc0e965cd0573c827"} Oct 30 18:16:10.323557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:10.323391 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" event={"ID":"bfa04282-d984-44f3-a442-655532f052af","Type":"ContainerStarted","Data":"d89bffed60190a9ab95494b5eb00715d7feeba6d2461e4ea434ff8272e3f7f81"} Oct 30 18:16:10.369483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:10.369429 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" podStartSLOduration=1.369411331 podStartE2EDuration="1.369411331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:10.366980677 +0000 UTC m=+1069.416025591" watchObservedRunningTime="2026-10-30 18:16:10.369411331 +0000 UTC m=+1069.418456246" Oct 30 18:16:10.419800 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:10.419771 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:10.421503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:10.421474 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:16:10.421630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:10.421529 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:16:10.669349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:10.668687 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xhh6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:16:10.669349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:10.668745 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:16:11.420608 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:11.420570 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.421091 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:11.420655 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:16:11.666288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:11.665669 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xhh6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.666288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:11.665770 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:16:12.419528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:12.419495 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.419696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:12.419569 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:16:12.671895 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:12.671797 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xhh6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.672506 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:12.672456 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:16:13.420141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:13.420103 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.420457 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:13.420429 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:16:13.665808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:13.665772 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xhh6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.666102 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:13.666071 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:16:14.419980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:14.419947 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.420587 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:14.420559 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:16:14.665475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:14.665434 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-xhh6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.665694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:14.665504 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:16:15.419651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:15.419617 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.419840 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:15.419681 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:16:15.980929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:15.980882 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:15.981837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:15.981815 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:15.982938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:15.982920 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:16:16.419605 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:16.419571 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.419824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:16.419624 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 18:16:17.400079 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.400048 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc"] Oct 30 18:16:17.402934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.402912 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.417830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.417809 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:16:17.417947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.417868 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:16:17.417947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.417868 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:16:17.417947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.417898 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4jrp2\"" Oct 30 18:16:17.424714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.424692 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:17.425257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.425233 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:17.426203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.426186 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:16:17.434960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.434939 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc"] Oct 30 18:16:17.505332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.505292 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6g5q\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-kube-api-access-h6g5q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.505514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.505438 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.505514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.505470 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.505514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.505497 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.505687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.505590 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.505687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.505675 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.505853 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.505826 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/074da152-8cdb-46c0-bf73-d800dd4355ff-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.607081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.607052 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6g5q\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-kube-api-access-h6g5q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.607275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.607117 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.607275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.607139 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.607275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.607159 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.607275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.607204 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.607481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.607255 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.607481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.607356 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/074da152-8cdb-46c0-bf73-d800dd4355ff-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.608008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.607978 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.609854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.609826 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.609938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.609870 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/074da152-8cdb-46c0-bf73-d800dd4355ff-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.609938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.609891 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.610022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.610006 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.626630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.626600 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6g5q\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-kube-api-access-h6g5q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.626747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.626706 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-d74mc\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:17.715483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:17.715402 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:18.042360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:18.042333 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc"] Oct 30 18:16:18.043378 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:16:18.043357 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074da152_8cdb_46c0_bf73_d800dd4355ff.slice/crio-6b5bf6be609269a447e123e54e5eea230fdaed3c9ab92ca493fcdf92406b9c40 WatchSource:0}: Error finding container 6b5bf6be609269a447e123e54e5eea230fdaed3c9ab92ca493fcdf92406b9c40: Status 404 returned error can't find the container with id 6b5bf6be609269a447e123e54e5eea230fdaed3c9ab92ca493fcdf92406b9c40 Oct 30 18:16:18.045809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:18.045772 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:18.045914 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:18.045838 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:16:18.416643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:18.416605 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" event={"ID":"074da152-8cdb-46c0-bf73-d800dd4355ff","Type":"ContainerStarted","Data":"72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f"} Oct 30 18:16:18.417071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:18.416668 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" event={"ID":"074da152-8cdb-46c0-bf73-d800dd4355ff","Type":"ContainerStarted","Data":"6b5bf6be609269a447e123e54e5eea230fdaed3c9ab92ca493fcdf92406b9c40"} Oct 30 18:16:18.417071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:18.416805 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:18.485742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:18.485689 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" podStartSLOduration=1.485671776 podStartE2EDuration="1.485671776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:18.470937957 +0000 UTC m=+1077.519982879" watchObservedRunningTime="2026-10-30 18:16:18.485671776 +0000 UTC m=+1077.534716689" Oct 30 18:16:19.425560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:19.425532 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 18:16:19.870658 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:16:19.870623 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:16:23.081293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.077526 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.098287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.094346 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.098287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.094667 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 30 18:16:23.247287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.246362 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.332346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.331353 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.332346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.331738 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 18:16:23.455290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.452376 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:23.482419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.482374 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.521289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.520350 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:23.525366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.525333 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 18:16:23.796347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:23.794382 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.002287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.000650 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 18:16:24.079328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.077689 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.108299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.079911 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:16:24.131162 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.124942 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 18:16:24.141293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.131467 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:16:24.375511 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.375411 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:16:24.439449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.375854 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.439449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.419896 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:16:24.439449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.419959 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:16:24.439449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.375894 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.442379 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.440834 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:16:24.455091 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.455052 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:24.462415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.462392 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:16:24.516289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.515976 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 18:16:24.520288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.516802 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:16:24.550343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.539821 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 18:16:24.552299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.552252 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:16:24.553391 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.553363 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.562284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.558869 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:16:24.635438 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.592356 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:24.741454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:24.741412 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.105301 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.049039 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 18:16:25.268846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.268751 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:25.269457 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.269425 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.269884 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.127822 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.384498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.087110 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.461880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.461840 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.519758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.470328 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 30 18:16:25.520007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.484469 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:16:25.528290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.489816 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.557293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.489853 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 18:16:25.557293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.519367 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.557293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.546792 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 18:16:25.557293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.552383 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.557293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.553653 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 18:16:25.601316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.596343 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.601316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.598614 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:16:25.614141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.613883 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 18:16:25.749435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.749394 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" podUID="da01fa56-122a-490d-999b-e2c20b957e4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:25.759291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:25.750560 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:16:26.161382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.161337 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.380286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.374350 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.388494 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.381237 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 18:16:26.413337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.411610 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 30 18:16:26.434117 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.424500 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 18:16:26.459285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.459034 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 18:16:26.495286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.483543 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 18:16:26.519409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.519382 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn" Oct 30 18:16:26.520053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.520020 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Oct 30 18:16:26.554657 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.554619 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.555666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.555636 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 18:16:26.625620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.625593 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 18:16:26.634977 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.634928 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:26.686561 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.686491 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 18:16:26.730465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.730438 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 18:16:26.738536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:26.738515 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 18:16:27.187434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:27.187405 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 18:16:27.187858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:27.187839 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 18:16:27.189536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:27.189517 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 18:16:44.955113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:16:44.955072 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:16:49.871922 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:16:49.871885 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:17:19.873071 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:17:19.873036 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:17:23.505197 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.505161 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl"] Oct 30 18:17:23.510252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.510228 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:23.529140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.529106 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl"] Oct 30 18:17:23.542774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.542747 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8dc43c\"" Oct 30 18:17:23.599056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.598684 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8dc43c-dockercfg-cmb4d\"" Oct 30 18:17:23.606061 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.606028 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162012bf-f307-453d-85e3-82a4c15eff8a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl\" (UID: \"162012bf-f307-453d-85e3-82a4c15eff8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:23.606655 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.606620 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97wf\" (UniqueName: \"kubernetes.io/projected/162012bf-f307-453d-85e3-82a4c15eff8a-kube-api-access-z97wf\") pod \"payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl\" (UID: \"162012bf-f307-453d-85e3-82a4c15eff8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:23.708608 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.708573 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162012bf-f307-453d-85e3-82a4c15eff8a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl\" (UID: \"162012bf-f307-453d-85e3-82a4c15eff8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:23.708791 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.708697 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97wf\" (UniqueName: \"kubernetes.io/projected/162012bf-f307-453d-85e3-82a4c15eff8a-kube-api-access-z97wf\") pod \"payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl\" (UID: \"162012bf-f307-453d-85e3-82a4c15eff8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:23.709317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.709287 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162012bf-f307-453d-85e3-82a4c15eff8a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl\" (UID: \"162012bf-f307-453d-85e3-82a4c15eff8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:23.728559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.728529 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97wf\" (UniqueName: \"kubernetes.io/projected/162012bf-f307-453d-85e3-82a4c15eff8a-kube-api-access-z97wf\") pod \"payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl\" (UID: \"162012bf-f307-453d-85e3-82a4c15eff8a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:23.928439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:23.928406 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:24.230710 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:24.230683 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl"] Oct 30 18:17:24.232257 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:17:24.232221 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162012bf_f307_453d_85e3_82a4c15eff8a.slice/crio-00f13341f4ef17c9cc0b3d84c3805452c66cd9493d6fcdb5aece2e5422157c84 WatchSource:0}: Error finding container 00f13341f4ef17c9cc0b3d84c3805452c66cd9493d6fcdb5aece2e5422157c84: Status 404 returned error can't find the container with id 00f13341f4ef17c9cc0b3d84c3805452c66cd9493d6fcdb5aece2e5422157c84 Oct 30 18:17:24.632628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:24.632590 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" event={"ID":"162012bf-f307-453d-85e3-82a4c15eff8a","Type":"ContainerStarted","Data":"446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa"} Oct 30 18:17:24.633112 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:24.632652 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" event={"ID":"162012bf-f307-453d-85e3-82a4c15eff8a","Type":"ContainerStarted","Data":"00f13341f4ef17c9cc0b3d84c3805452c66cd9493d6fcdb5aece2e5422157c84"} Oct 30 18:17:24.633112 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:24.632770 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:24.673924 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:24.673881 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" podStartSLOduration=1.67386938 podStartE2EDuration="1.67386938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:24.671205653 +0000 UTC m=+1143.720250564" watchObservedRunningTime="2026-10-30 18:17:24.67386938 +0000 UTC m=+1143.722914292" Oct 30 18:17:30.024392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:30.024356 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:17:35.647634 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:17:35.647605 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:17:49.869677 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:17:49.869638 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:18:08.088018 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:18:08.087982 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:18:19.872079 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:18:19.872038 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:18:21.195508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:18:21.195464 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:18:21.202598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:18:21.202571 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:18:38.942632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:18:38.942543 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:18:49.870460 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:18:49.870424 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:18:59.650866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:18:59.650833 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl"] Oct 30 18:19:04.651496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:04.651441 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" podUID="162012bf-f307-453d-85e3-82a4c15eff8a" containerName="payload-processing" containerID="cri-o://446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa" gracePeriod=25 Oct 30 18:19:04.918315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:04.918247 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:19:04.998544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:04.998512 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/162012bf-f307-453d-85e3-82a4c15eff8a-kube-api-access-z97wf\" (UniqueName: \"kubernetes.io/projected/162012bf-f307-453d-85e3-82a4c15eff8a-kube-api-access-z97wf\") pod \"162012bf-f307-453d-85e3-82a4c15eff8a\" (UID: \"162012bf-f307-453d-85e3-82a4c15eff8a\") " Oct 30 18:19:04.998846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:04.998824 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/162012bf-f307-453d-85e3-82a4c15eff8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162012bf-f307-453d-85e3-82a4c15eff8a-plugins-config-volume\") pod \"162012bf-f307-453d-85e3-82a4c15eff8a\" (UID: \"162012bf-f307-453d-85e3-82a4c15eff8a\") " Oct 30 18:19:04.999249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:04.999223 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/162012bf-f307-453d-85e3-82a4c15eff8a-plugins-config-volume" pod "162012bf-f307-453d-85e3-82a4c15eff8a" (UID: "162012bf-f307-453d-85e3-82a4c15eff8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:04.999487 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:04.999464 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/162012bf-f307-453d-85e3-82a4c15eff8a-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.000897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.000872 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162012bf-f307-453d-85e3-82a4c15eff8a-kube-api-access-z97wf" pod "162012bf-f307-453d-85e3-82a4c15eff8a" (UID: "162012bf-f307-453d-85e3-82a4c15eff8a"). InnerVolumeSpecName "kube-api-access-z97wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:05.101135 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.101106 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97wf\" (UniqueName: \"kubernetes.io/projected/162012bf-f307-453d-85e3-82a4c15eff8a-kube-api-access-z97wf\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:19:05.368946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.368918 4862 generic.go:356] "Generic (PLEG): container finished" podID="162012bf-f307-453d-85e3-82a4c15eff8a" containerID="446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa" exitCode=0 Oct 30 18:19:05.369116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.368960 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" Oct 30 18:19:05.369116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.369002 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" event={"ID":"162012bf-f307-453d-85e3-82a4c15eff8a","Type":"ContainerDied","Data":"446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa"} Oct 30 18:19:05.369116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.369056 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl" event={"ID":"162012bf-f307-453d-85e3-82a4c15eff8a","Type":"ContainerDied","Data":"00f13341f4ef17c9cc0b3d84c3805452c66cd9493d6fcdb5aece2e5422157c84"} Oct 30 18:19:05.369116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.369085 4862 scope.go:122] "RemoveContainer" containerID="446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa" Oct 30 18:19:05.382193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.382176 4862 scope.go:122] "RemoveContainer" containerID="446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa" Oct 30 18:19:05.382519 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:19:05.382501 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa\": container with ID starting with 446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa not found: ID does not exist" containerID="446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa" Oct 30 18:19:05.382577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.382526 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa"} err="failed to get container status \"446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa\": rpc error: code = NotFound desc = could not find container \"446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa\": container with ID starting with 446e4a5a81fe8b36fbd2cae444c2d687b37916f6a990ef30ea54400704715dfa not found: ID does not exist" Oct 30 18:19:05.420042 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.420015 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl"] Oct 30 18:19:05.432499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:05.432474 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8dc43c-5c757b7f8b-hpdrl"] Oct 30 18:19:07.065186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:07.065151 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="162012bf-f307-453d-85e3-82a4c15eff8a" path="/var/lib/kubelet/pods/162012bf-f307-453d-85e3-82a4c15eff8a/volumes" Oct 30 18:19:19.871637 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:19:19.871598 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:19:19.930051 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:19.930017 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:19:49.757456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.757420 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj"] Oct 30 18:19:49.758324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.758300 4862 cpu_manager.go:414] "containerMap: removing container" podUID="162012bf-f307-453d-85e3-82a4c15eff8a" containerName="payload-processing" Oct 30 18:19:49.758324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.758321 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="162012bf-f307-453d-85e3-82a4c15eff8a" containerName="payload-processing" Oct 30 18:19:49.758473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.758462 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="162012bf-f307-453d-85e3-82a4c15eff8a" containerName="payload-processing" Oct 30 18:19:49.762221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.762201 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.781503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.781474 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-podinfo\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.781639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.781532 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-token\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.781639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.781561 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istiod-ca-cert\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.781639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.781611 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx7mz\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-kube-api-access-nx7mz\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.781826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.781662 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-certs\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.781826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.781696 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-credential-socket\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.781826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.781756 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-envoy\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.782009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.781979 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-data\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.782086 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.782065 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-socket\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.793806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.793781 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-011d6f-openshift-default-dockercfg-tnjvg\"" Oct 30 18:19:49.797235 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.797212 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj"] Oct 30 18:19:49.873289 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:19:49.872861 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:19:49.883711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.883685 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-data\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.883868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.883738 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-socket\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.883868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.883773 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-podinfo\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.883868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.883800 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-token\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.883868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.883825 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istiod-ca-cert\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.884092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.883872 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx7mz\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-kube-api-access-nx7mz\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.884092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.883917 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-certs\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.884092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.883950 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-credential-socket\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.884092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.884000 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-envoy\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.884398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.884373 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-socket\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.884504 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.884462 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-certs\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.884759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.884590 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-credential-socket\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.884995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.884813 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-data\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.884995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.884896 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istiod-ca-cert\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.886910 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.886885 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-envoy\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.887387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.887365 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-podinfo\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.916978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.916914 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx7mz\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-kube-api-access-nx7mz\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:49.940540 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:49.940512 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-token\") pod \"e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:50.078594 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:50.078561 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:50.413614 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:19:50.413578 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4d1c63_04da_46e0_953e_40e5ef9663e1.slice/crio-4728fc7aaf4c9960b49f2274e1b04bcef7dbbfa400ecaa4ddb71d27b5b3e6ebe WatchSource:0}: Error finding container 4728fc7aaf4c9960b49f2274e1b04bcef7dbbfa400ecaa4ddb71d27b5b3e6ebe: Status 404 returned error can't find the container with id 4728fc7aaf4c9960b49f2274e1b04bcef7dbbfa400ecaa4ddb71d27b5b3e6ebe Oct 30 18:19:50.414202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:50.414130 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj"] Oct 30 18:19:50.417546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:50.417038 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:50.417546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:50.417122 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:50.417546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:50.417168 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:19:50.700045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:50.699957 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" event={"ID":"1a4d1c63-04da-46e0-953e-40e5ef9663e1","Type":"ContainerStarted","Data":"c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201"} Oct 30 18:19:50.700045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:50.700010 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" event={"ID":"1a4d1c63-04da-46e0-953e-40e5ef9663e1","Type":"ContainerStarted","Data":"4728fc7aaf4c9960b49f2274e1b04bcef7dbbfa400ecaa4ddb71d27b5b3e6ebe"} Oct 30 18:19:50.753723 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:50.753674 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" podStartSLOduration=1.7536612969999998 podStartE2EDuration="1.753661297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:50.739917866 +0000 UTC m=+1289.788962779" watchObservedRunningTime="2026-10-30 18:19:50.753661297 +0000 UTC m=+1289.802706201" Oct 30 18:19:51.079245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:51.079203 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:51.085778 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:51.085750 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:51.706922 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:51.706891 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:51.707812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:51.707788 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:19:52.655892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:52.655857 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7619-4e036a"] Oct 30 18:19:52.660015 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:52.659997 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7619-4e036a" Oct 30 18:19:52.674671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:52.674642 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7619-4e036a"] Oct 30 18:19:52.682404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:52.682371 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r2spd\"" Oct 30 18:19:52.722491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:52.722460 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxfl\" (UniqueName: \"kubernetes.io/projected/24da03ae-669f-42de-af11-d802768f01b5-kube-api-access-cbxfl\") pod \"test-curl-7619-4e036a\" (UID: \"24da03ae-669f-42de-af11-d802768f01b5\") " pod="opendatahub/test-curl-7619-4e036a" Oct 30 18:19:52.829284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:52.827530 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxfl\" (UniqueName: \"kubernetes.io/projected/24da03ae-669f-42de-af11-d802768f01b5-kube-api-access-cbxfl\") pod \"test-curl-7619-4e036a\" (UID: \"24da03ae-669f-42de-af11-d802768f01b5\") " pod="opendatahub/test-curl-7619-4e036a" Oct 30 18:19:52.864607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:52.864578 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxfl\" (UniqueName: \"kubernetes.io/projected/24da03ae-669f-42de-af11-d802768f01b5-kube-api-access-cbxfl\") pod \"test-curl-7619-4e036a\" (UID: \"24da03ae-669f-42de-af11-d802768f01b5\") " pod="opendatahub/test-curl-7619-4e036a" Oct 30 18:19:52.974657 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:52.974575 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7619-4e036a" Oct 30 18:19:53.285398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:53.285377 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7619-4e036a"] Oct 30 18:19:53.286519 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:19:53.286497 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24da03ae_669f_42de_af11_d802768f01b5.slice/crio-9d2c8430cb712b85b7ff968f67e65b1365806a5f3ab5a4d9753b6e2f8dbc212a WatchSource:0}: Error finding container 9d2c8430cb712b85b7ff968f67e65b1365806a5f3ab5a4d9753b6e2f8dbc212a: Status 404 returned error can't find the container with id 9d2c8430cb712b85b7ff968f67e65b1365806a5f3ab5a4d9753b6e2f8dbc212a Oct 30 18:19:53.720720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:53.720689 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7619-4e036a" event={"ID":"24da03ae-669f-42de-af11-d802768f01b5","Type":"ContainerStarted","Data":"9d2c8430cb712b85b7ff968f67e65b1365806a5f3ab5a4d9753b6e2f8dbc212a"} Oct 30 18:19:54.727950 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:54.727919 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7619-4e036a" event={"ID":"24da03ae-669f-42de-af11-d802768f01b5","Type":"ContainerStarted","Data":"0f06d39490641a995a6c4908441c2b7ce95e53f4ed6199177c64cba961375581"} Oct 30 18:19:54.767967 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:19:54.767907 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7619-4e036a" podStartSLOduration=2.110852648 podStartE2EDuration="2.767888295s" totalImagesPullingTime="657.035647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:54.76501076 +0000 UTC m=+1293.814055674" watchObservedRunningTime="2026-10-30 18:19:54.767888295 +0000 UTC m=+1293.816933218" Oct 30 18:20:05.655538 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:05.655458 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7619-4e036a"] Oct 30 18:20:05.656059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:05.655666 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7619-4e036a" podUID="24da03ae-669f-42de-af11-d802768f01b5" containerName="test-curl-7619-4e036a" containerID="cri-o://0f06d39490641a995a6c4908441c2b7ce95e53f4ed6199177c64cba961375581" gracePeriod=2 Oct 30 18:20:05.660892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:05.660864 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7619-4e036a"] Oct 30 18:20:05.669183 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:20:05.669157 4862 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7619-4e036a\" is forbidden: User \"system:node:ip-10-0-128-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-246.ec2.internal' and this object" podUID="24da03ae-669f-42de-af11-d802768f01b5" pod="opendatahub/test-curl-7619-4e036a" Oct 30 18:20:07.800137 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.800103 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph"] Oct 30 18:20:07.801133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.801109 4862 cpu_manager.go:414] "containerMap: removing container" podUID="24da03ae-669f-42de-af11-d802768f01b5" containerName="test-curl-7619-4e036a" Oct 30 18:20:07.801133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.801132 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24da03ae-669f-42de-af11-d802768f01b5" containerName="test-curl-7619-4e036a" Oct 30 18:20:07.801361 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.801336 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="24da03ae-669f-42de-af11-d802768f01b5" containerName="test-curl-7619-4e036a" Oct 30 18:20:07.805814 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.805797 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:07.821581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.821555 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph"] Oct 30 18:20:07.826579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.826521 4862 generic.go:356] "Generic (PLEG): container finished" podID="24da03ae-669f-42de-af11-d802768f01b5" containerID="0f06d39490641a995a6c4908441c2b7ce95e53f4ed6199177c64cba961375581" exitCode=137 Oct 30 18:20:07.856507 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.856467 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-de14f2\"" Oct 30 18:20:07.856671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.856552 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-de14f2-dockercfg-hc4qm\"" Oct 30 18:20:07.857388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.857355 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lkbp\" (UniqueName: \"kubernetes.io/projected/32fb59c9-17a1-4bbd-a374-de05dba686fa-kube-api-access-4lkbp\") pod \"payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph\" (UID: \"32fb59c9-17a1-4bbd-a374-de05dba686fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:07.857530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.857512 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fb59c9-17a1-4bbd-a374-de05dba686fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph\" (UID: \"32fb59c9-17a1-4bbd-a374-de05dba686fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:07.970203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.970165 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fb59c9-17a1-4bbd-a374-de05dba686fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph\" (UID: \"32fb59c9-17a1-4bbd-a374-de05dba686fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:07.970632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.970610 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lkbp\" (UniqueName: \"kubernetes.io/projected/32fb59c9-17a1-4bbd-a374-de05dba686fa-kube-api-access-4lkbp\") pod \"payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph\" (UID: \"32fb59c9-17a1-4bbd-a374-de05dba686fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:07.973568 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.973545 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fb59c9-17a1-4bbd-a374-de05dba686fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph\" (UID: \"32fb59c9-17a1-4bbd-a374-de05dba686fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:07.983024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.975612 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7619-4e036a" Oct 30 18:20:07.997829 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:07.997788 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lkbp\" (UniqueName: \"kubernetes.io/projected/32fb59c9-17a1-4bbd-a374-de05dba686fa-kube-api-access-4lkbp\") pod \"payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph\" (UID: \"32fb59c9-17a1-4bbd-a374-de05dba686fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:08.072390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.072356 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24da03ae-669f-42de-af11-d802768f01b5-kube-api-access-cbxfl\" (UniqueName: \"kubernetes.io/projected/24da03ae-669f-42de-af11-d802768f01b5-kube-api-access-cbxfl\") pod \"24da03ae-669f-42de-af11-d802768f01b5\" (UID: \"24da03ae-669f-42de-af11-d802768f01b5\") " Oct 30 18:20:08.074552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.074522 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24da03ae-669f-42de-af11-d802768f01b5-kube-api-access-cbxfl" pod "24da03ae-669f-42de-af11-d802768f01b5" (UID: "24da03ae-669f-42de-af11-d802768f01b5"). InnerVolumeSpecName "kube-api-access-cbxfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:08.127575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.127547 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:08.174006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.173978 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbxfl\" (UniqueName: \"kubernetes.io/projected/24da03ae-669f-42de-af11-d802768f01b5-kube-api-access-cbxfl\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:20:08.454039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.454009 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph"] Oct 30 18:20:08.455518 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:20:08.455494 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fb59c9_17a1_4bbd_a374_de05dba686fa.slice/crio-5ed696454bc4f0cee1fe62c9babe8ccdcd2ea08ad4437e1bbf40cda9df2cc209 WatchSource:0}: Error finding container 5ed696454bc4f0cee1fe62c9babe8ccdcd2ea08ad4437e1bbf40cda9df2cc209: Status 404 returned error can't find the container with id 5ed696454bc4f0cee1fe62c9babe8ccdcd2ea08ad4437e1bbf40cda9df2cc209 Oct 30 18:20:08.835792 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.835769 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7619-4e036a" Oct 30 18:20:08.836183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.835826 4862 scope.go:122] "RemoveContainer" containerID="0f06d39490641a995a6c4908441c2b7ce95e53f4ed6199177c64cba961375581" Oct 30 18:20:08.842885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.842851 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" event={"ID":"32fb59c9-17a1-4bbd-a374-de05dba686fa","Type":"ContainerStarted","Data":"efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c"} Oct 30 18:20:08.844410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.842914 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" event={"ID":"32fb59c9-17a1-4bbd-a374-de05dba686fa","Type":"ContainerStarted","Data":"5ed696454bc4f0cee1fe62c9babe8ccdcd2ea08ad4437e1bbf40cda9df2cc209"} Oct 30 18:20:08.844410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.843396 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:08.881725 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:08.881675 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" podStartSLOduration=1.8816606249999999 podStartE2EDuration="1.881660625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:08.881025883 +0000 UTC m=+1307.930070786" watchObservedRunningTime="2026-10-30 18:20:08.881660625 +0000 UTC m=+1307.930705554" Oct 30 18:20:08.894537 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:20:08.894506 4862 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7619-4e036a\" is forbidden: User \"system:node:ip-10-0-128-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-246.ec2.internal' and this object" podUID="24da03ae-669f-42de-af11-d802768f01b5" pod="opendatahub/test-curl-7619-4e036a" Oct 30 18:20:09.064111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:09.064079 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24da03ae-669f-42de-af11-d802768f01b5" path="/var/lib/kubelet/pods/24da03ae-669f-42de-af11-d802768f01b5/volumes" Oct 30 18:20:09.425787 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:09.425252 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:20:10.035628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:10.035598 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz"] Oct 30 18:20:10.046768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:10.046742 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4163c57f-d69bz"] Oct 30 18:20:11.065532 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:11.065501 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63b5e09-2f5e-4549-a13d-34632b46120d" path="/var/lib/kubelet/pods/c63b5e09-2f5e-4549-a13d-34632b46120d/volumes" Oct 30 18:20:19.853899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:19.853868 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:20:19.875811 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:20:19.875779 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:20:27.093954 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.093924 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm"] Oct 30 18:20:27.097737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.097722 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.125198 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.125169 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-de14f2-serving-cert\"" Oct 30 18:20:27.129382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.129353 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm"] Oct 30 18:20:27.158640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.158610 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-metrics-certs\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.158795 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.158754 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz488\" (UniqueName: \"kubernetes.io/projected/c2b3e467-726f-4625-8669-0ecefe4ae641-kube-api-access-sz488\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.158945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.158923 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.158996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.158964 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b3e467-726f-4625-8669-0ecefe4ae641-openshift-service-ca\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.260813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.260776 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-metrics-certs\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.260982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.260854 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz488\" (UniqueName: \"kubernetes.io/projected/c2b3e467-726f-4625-8669-0ecefe4ae641-kube-api-access-sz488\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.260982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.260943 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.260982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.260964 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b3e467-726f-4625-8669-0ecefe4ae641-openshift-service-ca\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.261192 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:20:27.261015 4862 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-serving-cert: secret "maas-api-e2e-api-b-de14f2-serving-cert" not found Oct 30 18:20:27.261192 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:20:27.261101 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls podName:c2b3e467-726f-4625-8669-0ecefe4ae641 nodeName:}" failed. No retries permitted until 2026-10-30 18:20:27.761079972 +0000 UTC m=+1326.810124877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls") pod "maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" (UID: "c2b3e467-726f-4625-8669-0ecefe4ae641") : secret "maas-api-e2e-api-b-de14f2-serving-cert" not found Oct 30 18:20:27.261698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.261673 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b3e467-726f-4625-8669-0ecefe4ae641-openshift-service-ca\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.263899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.263875 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-metrics-certs\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.282209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.282181 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz488\" (UniqueName: \"kubernetes.io/projected/c2b3e467-726f-4625-8669-0ecefe4ae641-kube-api-access-sz488\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.767800 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.767771 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:27.770346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:27.770319 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls\") pod \"maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:28.010575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:28.010539 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:28.318397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:28.318357 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm"] Oct 30 18:20:28.319074 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:20:28.319037 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b3e467_726f_4625_8669_0ecefe4ae641.slice/crio-b13881233251eb6ffcf03595805076cfb6a9ca7e2b480ca2581e5f17224965ff WatchSource:0}: Error finding container b13881233251eb6ffcf03595805076cfb6a9ca7e2b480ca2581e5f17224965ff: Status 404 returned error can't find the container with id b13881233251eb6ffcf03595805076cfb6a9ca7e2b480ca2581e5f17224965ff Oct 30 18:20:28.321095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:28.321050 4862 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:20:28.978861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:28.978780 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" event={"ID":"c2b3e467-726f-4625-8669-0ecefe4ae641","Type":"ContainerStarted","Data":"dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1"} Oct 30 18:20:28.978861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:28.978832 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" event={"ID":"c2b3e467-726f-4625-8669-0ecefe4ae641","Type":"ContainerStarted","Data":"b13881233251eb6ffcf03595805076cfb6a9ca7e2b480ca2581e5f17224965ff"} Oct 30 18:20:28.979054 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:28.978891 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:29.019963 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:29.019915 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" podStartSLOduration=1.660029559 podStartE2EDuration="2.019902536s" totalImagesPullingTime="359.872977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:29.019229439 +0000 UTC m=+1328.068274354" watchObservedRunningTime="2026-10-30 18:20:29.019902536 +0000 UTC m=+1328.068947485" Oct 30 18:20:29.909331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:29.909293 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls"] Oct 30 18:20:29.914424 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:29.914399 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:29.932536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:29.932510 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f826dd-openshift-default-dockercfg-5b2cn\"" Oct 30 18:20:29.948790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:29.948765 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls"] Oct 30 18:20:30.000818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.000788 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-certs\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.000986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.000868 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-token\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.001031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.000987 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-socket\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.001031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.001015 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-envoy\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.001096 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.001038 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-credential-socket\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.001153 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.001137 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcm4\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-kube-api-access-glcm4\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.001221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.001209 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-data\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.001328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.001314 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istiod-ca-cert\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.001407 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.001391 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-podinfo\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.104579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.104151 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-token\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.104579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.104301 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-socket\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.104579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.104336 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-envoy\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.104579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.104367 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-credential-socket\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.104579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.104467 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcm4\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-kube-api-access-glcm4\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.104579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.104545 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-data\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.105048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.104659 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istiod-ca-cert\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.105048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.104748 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-podinfo\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.105048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.104779 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-certs\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.108801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.105326 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-certs\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.108801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.106233 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-socket\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.128319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.123908 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-data\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.128319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.124122 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-credential-socket\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.128319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.124762 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istiod-ca-cert\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.137564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.137526 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-envoy\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.146941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.146914 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcm4\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-kube-api-access-glcm4\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.148562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.148537 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-token\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.149640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.149600 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-podinfo\") pod \"e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.250701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.250622 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:30.597779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.597753 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls"] Oct 30 18:20:30.602640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.602602 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:30.602757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.602677 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:30.602757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.602710 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:30.995913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.995829 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" event={"ID":"ba535e0e-ea07-4d37-8c4c-747b060a62dd","Type":"ContainerStarted","Data":"21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379"} Oct 30 18:20:30.995913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:30.995885 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" event={"ID":"ba535e0e-ea07-4d37-8c4c-747b060a62dd","Type":"ContainerStarted","Data":"dda31befb0c5cd5dbd79f8b3eec69aaacb3369ffd1ea94b773eba2e69d769b48"} Oct 30 18:20:31.050846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:31.050790 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" podStartSLOduration=2.050776502 podStartE2EDuration="2.050776502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:31.03755696 +0000 UTC m=+1330.086601873" watchObservedRunningTime="2026-10-30 18:20:31.050776502 +0000 UTC m=+1330.099821432" Oct 30 18:20:31.251275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:31.251172 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:31.253075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:31.253050 4862 patch_prober.go:29] interesting pod/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:20:31.253323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:31.253293 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:20:32.255889 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:32.255860 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:33.008460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:33.008426 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:33.009337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:33.009318 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:20:34.992559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:34.992530 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:20:42.792984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:42.792933 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:20:49.873424 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:20:49.873333 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:20:52.748394 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:52.748361 4862 patch_prober.go:29] interesting pod/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:52.748872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:52.748438 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:54.160819 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:54.160785 4862 generic.go:356] "Generic (PLEG): container finished" podID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerID="c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201" exitCode=0 Oct 30 18:20:54.161246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:54.160855 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" event={"ID":"1a4d1c63-04da-46e0-953e-40e5ef9663e1","Type":"ContainerDied","Data":"c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201"} Oct 30 18:20:54.161246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:54.161238 4862 scope.go:122] "RemoveContainer" containerID="c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201" Oct 30 18:20:54.162780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:54.162754 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:54.162857 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:54.162825 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:54.162896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:54.162857 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:20:55.169401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:55.169367 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" event={"ID":"1a4d1c63-04da-46e0-953e-40e5ef9663e1","Type":"ContainerStarted","Data":"f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759"} Oct 30 18:20:56.079001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:56.078961 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:20:56.079563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:56.079159 4862 patch_prober.go:29] interesting pod/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:20:56.079563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:56.079212 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:20:57.080465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:57.080433 4862 patch_prober.go:29] interesting pod/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:20:57.081092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:57.081045 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:20:58.079748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:58.079717 4862 patch_prober.go:29] interesting pod/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:20:58.079911 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:58.079766 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:20:59.079388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:59.079354 4862 patch_prober.go:29] interesting pod/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:20:59.079782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:20:59.079417 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:21:00.079706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:00.079673 4862 patch_prober.go:29] interesting pod/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:21:00.080117 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:00.079738 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 18:21:01.082837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:01.082810 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:21:01.083432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:01.083177 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:21:01.083810 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:01.083791 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:21:07.396200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:07.396150 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:21:18.010221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:18.010188 4862 patch_prober.go:29] interesting pod/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:21:18.010656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:18.010250 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:21:18.342178 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:18.342145 4862 generic.go:356] "Generic (PLEG): container finished" podID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerID="21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379" exitCode=0 Oct 30 18:21:18.342348 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:18.342217 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" event={"ID":"ba535e0e-ea07-4d37-8c4c-747b060a62dd","Type":"ContainerDied","Data":"21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379"} Oct 30 18:21:18.342599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:18.342585 4862 scope.go:122] "RemoveContainer" containerID="21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379" Oct 30 18:21:18.344033 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:18.344007 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:21:18.344096 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:18.344076 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:21:18.344132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:18.344107 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:21:19.355444 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:19.355401 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" event={"ID":"ba535e0e-ea07-4d37-8c4c-747b060a62dd","Type":"ContainerStarted","Data":"1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9"} Oct 30 18:21:19.884380 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:21:19.884347 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:21:20.253119 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:20.252545 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:21:20.253119 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:20.252658 4862 patch_prober.go:29] interesting pod/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:21:20.253119 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:20.252714 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:21:21.252288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:21.251483 4862 patch_prober.go:29] interesting pod/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:21:21.252288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:21.251544 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:21:21.471435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:21.471405 4862 scope.go:122] "RemoveContainer" containerID="33bd4f40f1a112d1145da39b82d7a32f8fdf60d4906159942e96ca70eaa191b0" Oct 30 18:21:22.253288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:22.252627 4862 patch_prober.go:29] interesting pod/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:21:22.253288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:22.252700 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:21:23.253290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:23.252644 4862 patch_prober.go:29] interesting pod/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:21:23.253290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:23.252709 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:21:24.251673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:24.251632 4862 patch_prober.go:29] interesting pod/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:21:24.251959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:24.251929 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:21:25.255544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:25.255516 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:21:25.255917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:25.255860 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:21:25.256610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:25.256596 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:21:49.888111 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:21:49.888023 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:21:58.265359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:58.265327 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw"] Oct 30 18:21:58.265791 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:21:58.265591 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" containerID="cri-o://b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089" gracePeriod=30 Oct 30 18:22:03.554288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.554240 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:22:03.655605 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.655578 4862 generic.go:356] "Generic (PLEG): container finished" podID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerID="b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089" exitCode=0 Oct 30 18:22:03.655785 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.655631 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" Oct 30 18:22:03.655785 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.655665 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" event={"ID":"0c4bce15-f17a-460c-acb5-b05073ec9eda","Type":"ContainerDied","Data":"b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089"} Oct 30 18:22:03.655785 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.655719 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw" event={"ID":"0c4bce15-f17a-460c-acb5-b05073ec9eda","Type":"ContainerDied","Data":"ddee0906d25b1f994538193ceb46fea9fb9abe222dfec2f50264b8ea620c644f"} Oct 30 18:22:03.655785 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.655747 4862 scope.go:122] "RemoveContainer" containerID="b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089" Oct 30 18:22:03.661040 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661021 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-kube-api-access-8qhfv\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-kube-api-access-8qhfv\") pod \"0c4bce15-f17a-460c-acb5-b05073ec9eda\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " Oct 30 18:22:03.661157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661092 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4bce15-f17a-460c-acb5-b05073ec9eda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4bce15-f17a-460c-acb5-b05073ec9eda-istiod-ca-cert\") pod \"0c4bce15-f17a-460c-acb5-b05073ec9eda\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " Oct 30 18:22:03.661157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661115 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-data\") pod \"0c4bce15-f17a-460c-acb5-b05073ec9eda\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " Oct 30 18:22:03.661347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661330 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-certs\") pod \"0c4bce15-f17a-460c-acb5-b05073ec9eda\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " Oct 30 18:22:03.661444 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661425 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-podinfo\") pod \"0c4bce15-f17a-460c-acb5-b05073ec9eda\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " Oct 30 18:22:03.661518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661442 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-data" pod "0c4bce15-f17a-460c-acb5-b05073ec9eda" (UID: "0c4bce15-f17a-460c-acb5-b05073ec9eda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.661518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661479 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-envoy\") pod \"0c4bce15-f17a-460c-acb5-b05073ec9eda\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " Oct 30 18:22:03.661623 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661533 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-token\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-token\") pod \"0c4bce15-f17a-460c-acb5-b05073ec9eda\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " Oct 30 18:22:03.661623 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661537 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4bce15-f17a-460c-acb5-b05073ec9eda-istiod-ca-cert" pod "0c4bce15-f17a-460c-acb5-b05073ec9eda" (UID: "0c4bce15-f17a-460c-acb5-b05073ec9eda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:03.661729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661640 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-certs" pod "0c4bce15-f17a-460c-acb5-b05073ec9eda" (UID: "0c4bce15-f17a-460c-acb5-b05073ec9eda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.661729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661573 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-credential-socket\") pod \"0c4bce15-f17a-460c-acb5-b05073ec9eda\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " Oct 30 18:22:03.661729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661695 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-socket\") pod \"0c4bce15-f17a-460c-acb5-b05073ec9eda\" (UID: \"0c4bce15-f17a-460c-acb5-b05073ec9eda\") " Oct 30 18:22:03.661849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661771 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-credential-socket" pod "0c4bce15-f17a-460c-acb5-b05073ec9eda" (UID: "0c4bce15-f17a-460c-acb5-b05073ec9eda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.661974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.661950 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-socket" pod "0c4bce15-f17a-460c-acb5-b05073ec9eda" (UID: "0c4bce15-f17a-460c-acb5-b05073ec9eda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.662794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.662668 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.662794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.662695 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.662794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.662710 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.662794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.662724 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4bce15-f17a-460c-acb5-b05073ec9eda-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.662794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.662737 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.663923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.663898 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-kube-api-access-8qhfv" pod "0c4bce15-f17a-460c-acb5-b05073ec9eda" (UID: "0c4bce15-f17a-460c-acb5-b05073ec9eda"). InnerVolumeSpecName "kube-api-access-8qhfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:03.664316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.664294 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-podinfo" pod "0c4bce15-f17a-460c-acb5-b05073ec9eda" (UID: "0c4bce15-f17a-460c-acb5-b05073ec9eda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:03.664398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.664384 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-token" pod "0c4bce15-f17a-460c-acb5-b05073ec9eda" (UID: "0c4bce15-f17a-460c-acb5-b05073ec9eda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:03.664434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.664405 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-envoy" pod "0c4bce15-f17a-460c-acb5-b05073ec9eda" (UID: "0c4bce15-f17a-460c-acb5-b05073ec9eda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:03.666112 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.666089 4862 scope.go:122] "RemoveContainer" containerID="b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089" Oct 30 18:22:03.666420 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:22:03.666399 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089\": container with ID starting with b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089 not found: ID does not exist" containerID="b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089" Oct 30 18:22:03.666516 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.666428 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089"} err="failed to get container status \"b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089\": rpc error: code = NotFound desc = could not find container \"b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089\": container with ID starting with b43841f0864ecce4265eff20a0e2edb39ee036ebbbe18f3e8e1ed6522aa02089 not found: ID does not exist" Oct 30 18:22:03.763997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.763970 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.763997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.763998 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.763997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.764007 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:03.764226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:03.764020 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qhfv\" (UniqueName: \"kubernetes.io/projected/0c4bce15-f17a-460c-acb5-b05073ec9eda-kube-api-access-8qhfv\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:04.002600 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:04.002569 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw"] Oct 30 18:22:04.014704 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:04.014683 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-th5rw"] Oct 30 18:22:05.063160 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:05.063121 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" path="/var/lib/kubelet/pods/0c4bce15-f17a-460c-acb5-b05073ec9eda/volumes" Oct 30 18:22:06.906135 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:06.906094 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:22:18.279085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:18.279048 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r"] Oct 30 18:22:18.279555 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:18.279271 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" containerID="cri-o://2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e" gracePeriod=30 Oct 30 18:22:19.872202 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:22:19.872165 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:22:19.969755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:19.969719 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph"] Oct 30 18:22:23.559396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.559372 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:22:23.745768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.745736 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-credential-socket\") pod \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " Oct 30 18:22:23.746229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.745792 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-kube-api-access-jrlq7\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-kube-api-access-jrlq7\") pod \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " Oct 30 18:22:23.746229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.745849 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-envoy\") pod \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " Oct 30 18:22:23.746229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.745875 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istiod-ca-cert\") pod \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " Oct 30 18:22:23.746229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746126 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-credential-socket" pod "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" (UID: "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.746563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746300 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-certs\") pod \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " Oct 30 18:22:23.746563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746393 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-data\") pod \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " Oct 30 18:22:23.746563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746516 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-socket\") pod \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " Oct 30 18:22:23.746716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746607 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-podinfo\") pod \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " Oct 30 18:22:23.746716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746618 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-certs" pod "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" (UID: "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.746716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746544 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istiod-ca-cert" pod "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" (UID: "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:23.746716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746683 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-token\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-token\") pod \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\" (UID: \"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b\") " Oct 30 18:22:23.746953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746815 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-data" pod "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" (UID: "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.746953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.746839 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-socket" pod "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" (UID: "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.747356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.747333 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.747482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.747415 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.747482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.747428 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.747610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.747442 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.747610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.747587 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.749084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.749057 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-kube-api-access-jrlq7" pod "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" (UID: "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b"). InnerVolumeSpecName "kube-api-access-jrlq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:23.749612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.749577 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-envoy" pod "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" (UID: "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:23.749698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.749627 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-token" pod "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" (UID: "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:23.749839 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.749806 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-podinfo" pod "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" (UID: "343b1bdb-a9bc-415c-bcf3-a3cf7de4200b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:23.819036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.818969 4862 generic.go:356] "Generic (PLEG): container finished" podID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerID="2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e" exitCode=0 Oct 30 18:22:23.819036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.819013 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" Oct 30 18:22:23.819183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.819052 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" event={"ID":"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b","Type":"ContainerDied","Data":"2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e"} Oct 30 18:22:23.819183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.819112 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r" event={"ID":"343b1bdb-a9bc-415c-bcf3-a3cf7de4200b","Type":"ContainerDied","Data":"cd6955b1d027adc4ceeb6410c703b7293946fa1bb729d1d01ee2aee9c953c465"} Oct 30 18:22:23.819183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.819137 4862 scope.go:122] "RemoveContainer" containerID="2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e" Oct 30 18:22:23.830755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.830737 4862 scope.go:122] "RemoveContainer" containerID="2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e" Oct 30 18:22:23.831012 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:22:23.830991 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e\": container with ID starting with 2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e not found: ID does not exist" containerID="2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e" Oct 30 18:22:23.831053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.831028 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e"} err="failed to get container status \"2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e\": rpc error: code = NotFound desc = could not find container \"2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e\": container with ID starting with 2774ba449c1f669a6d537e7f71d719b37d0c46506e1e095f95c25824c9ddd50e not found: ID does not exist" Oct 30 18:22:23.848481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.848455 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.848481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.848477 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.848481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.848487 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrlq7\" (UniqueName: \"kubernetes.io/projected/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-kube-api-access-jrlq7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.848733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.848496 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:23.871950 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.871920 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r"] Oct 30 18:22:23.875031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:23.875005 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-v9p8r"] Oct 30 18:22:24.970095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:24.970053 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" podUID="32fb59c9-17a1-4bbd-a374-de05dba686fa" containerName="payload-pre-processing" containerID="cri-o://efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c" gracePeriod=25 Oct 30 18:22:25.062381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.062355 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" path="/var/lib/kubelet/pods/343b1bdb-a9bc-415c-bcf3-a3cf7de4200b/volumes" Oct 30 18:22:25.243471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.243450 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:22:25.369025 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.368998 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fb59c9-17a1-4bbd-a374-de05dba686fa-kube-api-access-4lkbp\" (UniqueName: \"kubernetes.io/projected/32fb59c9-17a1-4bbd-a374-de05dba686fa-kube-api-access-4lkbp\") pod \"32fb59c9-17a1-4bbd-a374-de05dba686fa\" (UID: \"32fb59c9-17a1-4bbd-a374-de05dba686fa\") " Oct 30 18:22:25.369200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.369187 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fb59c9-17a1-4bbd-a374-de05dba686fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fb59c9-17a1-4bbd-a374-de05dba686fa-plugins-config-volume\") pod \"32fb59c9-17a1-4bbd-a374-de05dba686fa\" (UID: \"32fb59c9-17a1-4bbd-a374-de05dba686fa\") " Oct 30 18:22:25.369637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.369609 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fb59c9-17a1-4bbd-a374-de05dba686fa-plugins-config-volume" pod "32fb59c9-17a1-4bbd-a374-de05dba686fa" (UID: "32fb59c9-17a1-4bbd-a374-de05dba686fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:25.371315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.371240 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fb59c9-17a1-4bbd-a374-de05dba686fa-kube-api-access-4lkbp" pod "32fb59c9-17a1-4bbd-a374-de05dba686fa" (UID: "32fb59c9-17a1-4bbd-a374-de05dba686fa"). InnerVolumeSpecName "kube-api-access-4lkbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:25.470337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.470316 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lkbp\" (UniqueName: \"kubernetes.io/projected/32fb59c9-17a1-4bbd-a374-de05dba686fa-kube-api-access-4lkbp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:25.470337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.470338 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32fb59c9-17a1-4bbd-a374-de05dba686fa-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:22:25.835850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.835814 4862 generic.go:356] "Generic (PLEG): container finished" podID="32fb59c9-17a1-4bbd-a374-de05dba686fa" containerID="efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c" exitCode=0 Oct 30 18:22:25.836060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.835913 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" event={"ID":"32fb59c9-17a1-4bbd-a374-de05dba686fa","Type":"ContainerDied","Data":"efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c"} Oct 30 18:22:25.836060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.835996 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" event={"ID":"32fb59c9-17a1-4bbd-a374-de05dba686fa","Type":"ContainerDied","Data":"5ed696454bc4f0cee1fe62c9babe8ccdcd2ea08ad4437e1bbf40cda9df2cc209"} Oct 30 18:22:25.836060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.836037 4862 scope.go:122] "RemoveContainer" containerID="efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c" Oct 30 18:22:25.836249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.836234 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph" Oct 30 18:22:25.849643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.849626 4862 scope.go:122] "RemoveContainer" containerID="efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c" Oct 30 18:22:25.849905 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:22:25.849887 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c\": container with ID starting with efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c not found: ID does not exist" containerID="efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c" Oct 30 18:22:25.849967 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.849913 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c"} err="failed to get container status \"efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c\": rpc error: code = NotFound desc = could not find container \"efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c\": container with ID starting with efd483d1b990398b776c6bed514b07a351c90ad307c20c1fba4c86dde9b6068c not found: ID does not exist" Oct 30 18:22:25.888059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.888036 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph"] Oct 30 18:22:25.900071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:25.900048 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-de14f2-689d7c47c5-lnpph"] Oct 30 18:22:27.062770 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:27.062730 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fb59c9-17a1-4bbd-a374-de05dba686fa" path="/var/lib/kubelet/pods/32fb59c9-17a1-4bbd-a374-de05dba686fa/volumes" Oct 30 18:22:32.272285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:22:32.265432 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:22:49.869777 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:22:49.869744 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:23:01.420687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:01.420606 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm"] Oct 30 18:23:01.421317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:01.420850 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" podUID="c2b3e467-726f-4625-8669-0ecefe4ae641" containerName="maas-api" containerID="cri-o://dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1" gracePeriod=30 Oct 30 18:23:01.847905 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:01.846249 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:23:02.013287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.006955 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b3e467-726f-4625-8669-0ecefe4ae641-kube-api-access-sz488\" (UniqueName: \"kubernetes.io/projected/c2b3e467-726f-4625-8669-0ecefe4ae641-kube-api-access-sz488\") pod \"c2b3e467-726f-4625-8669-0ecefe4ae641\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " Oct 30 18:23:02.013287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.007021 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b3e467-726f-4625-8669-0ecefe4ae641-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b3e467-726f-4625-8669-0ecefe4ae641-openshift-service-ca\") pod \"c2b3e467-726f-4625-8669-0ecefe4ae641\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " Oct 30 18:23:02.013287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.007301 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-metrics-certs\") pod \"c2b3e467-726f-4625-8669-0ecefe4ae641\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " Oct 30 18:23:02.013287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.007364 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls\") pod \"c2b3e467-726f-4625-8669-0ecefe4ae641\" (UID: \"c2b3e467-726f-4625-8669-0ecefe4ae641\") " Oct 30 18:23:02.025923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.025523 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b3e467-726f-4625-8669-0ecefe4ae641-openshift-service-ca" pod "c2b3e467-726f-4625-8669-0ecefe4ae641" (UID: "c2b3e467-726f-4625-8669-0ecefe4ae641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:02.035763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.035719 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b3e467-726f-4625-8669-0ecefe4ae641-kube-api-access-sz488" pod "c2b3e467-726f-4625-8669-0ecefe4ae641" (UID: "c2b3e467-726f-4625-8669-0ecefe4ae641"). InnerVolumeSpecName "kube-api-access-sz488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:02.039091 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.039064 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls" pod "c2b3e467-726f-4625-8669-0ecefe4ae641" (UID: "c2b3e467-726f-4625-8669-0ecefe4ae641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:02.039397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.039369 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-metrics-certs" pod "c2b3e467-726f-4625-8669-0ecefe4ae641" (UID: "c2b3e467-726f-4625-8669-0ecefe4ae641"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:02.078562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.078522 4862 generic.go:356] "Generic (PLEG): container finished" podID="c2b3e467-726f-4625-8669-0ecefe4ae641" containerID="dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1" exitCode=0 Oct 30 18:23:02.078562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.078571 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" Oct 30 18:23:02.078862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.078573 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" event={"ID":"c2b3e467-726f-4625-8669-0ecefe4ae641","Type":"ContainerDied","Data":"dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1"} Oct 30 18:23:02.078862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.078704 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm" event={"ID":"c2b3e467-726f-4625-8669-0ecefe4ae641","Type":"ContainerDied","Data":"b13881233251eb6ffcf03595805076cfb6a9ca7e2b480ca2581e5f17224965ff"} Oct 30 18:23:02.078862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.078742 4862 scope.go:122] "RemoveContainer" containerID="dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1" Oct 30 18:23:02.090112 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.090096 4862 scope.go:122] "RemoveContainer" containerID="dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1" Oct 30 18:23:02.090435 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:23:02.090376 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1\": container with ID starting with dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1 not found: ID does not exist" containerID="dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1" Oct 30 18:23:02.090435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.090409 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1"} err="failed to get container status \"dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1\": rpc error: code = NotFound desc = could not find container \"dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1\": container with ID starting with dafcbdbbebeeffd6172db51458cfb5bb34dc87c5ad3676d7701dd5753288bda1 not found: ID does not exist" Oct 30 18:23:02.109752 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.109721 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz488\" (UniqueName: \"kubernetes.io/projected/c2b3e467-726f-4625-8669-0ecefe4ae641-kube-api-access-sz488\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:23:02.109752 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.109745 4862 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2b3e467-726f-4625-8669-0ecefe4ae641-openshift-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:23:02.109752 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.109756 4862 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-metrics-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:23:02.109914 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.109764 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2b3e467-726f-4625-8669-0ecefe4ae641-maas-api-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:23:02.132474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.132454 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm"] Oct 30 18:23:02.135676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:02.135658 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-de14f2-7cc7979b7-2b6pm"] Oct 30 18:23:03.062355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:03.062318 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b3e467-726f-4625-8669-0ecefe4ae641" path="/var/lib/kubelet/pods/c2b3e467-726f-4625-8669-0ecefe4ae641/volumes" Oct 30 18:23:19.873602 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:23:19.873560 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:23:21.250235 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:21.250205 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:23:21.259072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:21.259042 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:23:25.577286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:25.576731 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:23:42.376662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:42.376631 4862 generic.go:356] "Generic (PLEG): container finished" podID="58092d26-be96-4cc3-8584-cb9b729475dd" containerID="2d66e9706aa1ea0bf90ddbbcf034c575b50760bff701b351af348d0a982a8255" exitCode=1 Oct 30 18:23:42.377050 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:42.376706 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" event={"ID":"58092d26-be96-4cc3-8584-cb9b729475dd","Type":"ContainerDied","Data":"2d66e9706aa1ea0bf90ddbbcf034c575b50760bff701b351af348d0a982a8255"} Oct 30 18:23:42.377136 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:42.377118 4862 scope.go:122] "RemoveContainer" containerID="2d66e9706aa1ea0bf90ddbbcf034c575b50760bff701b351af348d0a982a8255" Oct 30 18:23:43.385704 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.385673 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" event={"ID":"58092d26-be96-4cc3-8584-cb9b729475dd","Type":"ContainerStarted","Data":"099d6e302701fa2b4961cea49854a14913b422f8e4d81b0e559d6e7fbaa65946"} Oct 30 18:23:43.386181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.385885 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:23:43.697311 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697202 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8"] Oct 30 18:23:43.697817 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697802 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c2b3e467-726f-4625-8669-0ecefe4ae641" containerName="maas-api" Oct 30 18:23:43.697859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697819 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b3e467-726f-4625-8669-0ecefe4ae641" containerName="maas-api" Oct 30 18:23:43.697859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697856 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" Oct 30 18:23:43.697928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697862 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" Oct 30 18:23:43.697928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697872 4862 cpu_manager.go:414] "containerMap: removing container" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" Oct 30 18:23:43.697928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697877 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" Oct 30 18:23:43.697928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697889 4862 cpu_manager.go:414] "containerMap: removing container" podUID="32fb59c9-17a1-4bbd-a374-de05dba686fa" containerName="payload-pre-processing" Oct 30 18:23:43.697928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697893 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fb59c9-17a1-4bbd-a374-de05dba686fa" containerName="payload-pre-processing" Oct 30 18:23:43.698078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.697994 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4bce15-f17a-460c-acb5-b05073ec9eda" containerName="istio-proxy" Oct 30 18:23:43.698078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.698001 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="343b1bdb-a9bc-415c-bcf3-a3cf7de4200b" containerName="istio-proxy" Oct 30 18:23:43.698078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.698009 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fb59c9-17a1-4bbd-a374-de05dba686fa" containerName="payload-pre-processing" Oct 30 18:23:43.698078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.698018 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b3e467-726f-4625-8669-0ecefe4ae641" containerName="maas-api" Oct 30 18:23:43.701392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.701374 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.723889 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.723862 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8"] Oct 30 18:23:43.732649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.732624 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a4f3f0-serving-cert\"" Oct 30 18:23:43.820632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.820599 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.820790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.820635 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-metrics-certs\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.820835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.820812 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjm52\" (UniqueName: \"kubernetes.io/projected/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-kube-api-access-zjm52\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.820871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.820842 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.922135 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.922104 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjm52\" (UniqueName: \"kubernetes.io/projected/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-kube-api-access-zjm52\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.922373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.922157 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.922373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.922300 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.922373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.922334 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-metrics-certs\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.922957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.922934 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.926899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.926865 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-metrics-certs\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.927366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.927345 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:43.973255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:43.973164 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjm52\" (UniqueName: \"kubernetes.io/projected/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-kube-api-access-zjm52\") pod \"maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:44.015898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:44.015857 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:44.323191 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:44.323168 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8"] Oct 30 18:23:44.324430 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:23:44.324401 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86e3b0b_0fe1_417e_b000_e3f84310f2f8.slice/crio-7085a93a4bc616b366ee7ed71185dcf43008f8282386b6cba7e200e1bb9b1cb6 WatchSource:0}: Error finding container 7085a93a4bc616b366ee7ed71185dcf43008f8282386b6cba7e200e1bb9b1cb6: Status 404 returned error can't find the container with id 7085a93a4bc616b366ee7ed71185dcf43008f8282386b6cba7e200e1bb9b1cb6 Oct 30 18:23:44.393287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:44.393243 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" event={"ID":"b86e3b0b-0fe1-417e-b000-e3f84310f2f8","Type":"ContainerStarted","Data":"7085a93a4bc616b366ee7ed71185dcf43008f8282386b6cba7e200e1bb9b1cb6"} Oct 30 18:23:45.401530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:45.401495 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" event={"ID":"b86e3b0b-0fe1-417e-b000-e3f84310f2f8","Type":"ContainerStarted","Data":"a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa"} Oct 30 18:23:45.402355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:45.401615 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:23:45.443981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:45.443919 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" podStartSLOduration=2.152627773 podStartE2EDuration="2.443902234s" totalImagesPullingTime="291.274461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:45.440883063 +0000 UTC m=+1524.489927977" watchObservedRunningTime="2026-10-30 18:23:45.443902234 +0000 UTC m=+1524.492947147" Oct 30 18:23:49.869827 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:23:49.869786 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:23:51.412069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:23:51.412045 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:24:02.324804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:24:02.324762 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:24:14.395634 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:24:14.395607 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jzvfr" Oct 30 18:24:19.914047 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:24:19.913962 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:24:30.911319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:24:30.911281 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:24:49.869741 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:24:49.869705 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:25:14.590503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:14.590467 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj"] Oct 30 18:25:14.591055 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:14.590712 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" containerID="cri-o://f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759" gracePeriod=30 Oct 30 18:25:16.085385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:16.085352 4862 patch_prober.go:29] interesting pod/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:25:16.085772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:16.085409 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:25:19.045717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:19.040026 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:25:19.924412 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:25:19.924322 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:25:19.971410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:19.971385 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:25:20.099064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.099027 4862 generic.go:356] "Generic (PLEG): container finished" podID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerID="f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759" exitCode=0 Oct 30 18:25:20.099474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.099092 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" Oct 30 18:25:20.099474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.099102 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" event={"ID":"1a4d1c63-04da-46e0-953e-40e5ef9663e1","Type":"ContainerDied","Data":"f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759"} Oct 30 18:25:20.099474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.099154 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj" event={"ID":"1a4d1c63-04da-46e0-953e-40e5ef9663e1","Type":"ContainerDied","Data":"4728fc7aaf4c9960b49f2274e1b04bcef7dbbfa400ecaa4ddb71d27b5b3e6ebe"} Oct 30 18:25:20.099474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.099183 4862 scope.go:122] "RemoveContainer" containerID="f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759" Oct 30 18:25:20.109281 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.109244 4862 scope.go:122] "RemoveContainer" containerID="c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201" Oct 30 18:25:20.118645 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.118628 4862 scope.go:122] "RemoveContainer" containerID="f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759" Oct 30 18:25:20.118900 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:25:20.118868 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759\": container with ID starting with f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759 not found: ID does not exist" containerID="f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759" Oct 30 18:25:20.118900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.118894 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759"} err="failed to get container status \"f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759\": rpc error: code = NotFound desc = could not find container \"f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759\": container with ID starting with f1deaf5848f6033483d781ae93b0849dc7d2dd358c1f5a50a890ce9d05c5f759 not found: ID does not exist" Oct 30 18:25:20.119052 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.118910 4862 scope.go:122] "RemoveContainer" containerID="c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201" Oct 30 18:25:20.119136 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:25:20.119121 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201\": container with ID starting with c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201 not found: ID does not exist" containerID="c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201" Oct 30 18:25:20.119184 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119144 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201"} err="failed to get container status \"c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201\": rpc error: code = NotFound desc = could not find container \"c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201\": container with ID starting with c2bd1519996347d4b9c0a26bb79ab7483c5f1733df5006fb829752b649279201 not found: ID does not exist" Oct 30 18:25:20.119222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119187 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-certs\") pod \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " Oct 30 18:25:20.119321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119305 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-token\") pod \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " Oct 30 18:25:20.119459 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119443 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-kube-api-access-nx7mz\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-kube-api-access-nx7mz\") pod \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " Oct 30 18:25:20.119546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119531 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-podinfo\") pod \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " Oct 30 18:25:20.119607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119561 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-data\") pod \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " Oct 30 18:25:20.119607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119598 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-credential-socket\") pod \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " Oct 30 18:25:20.119713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119631 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-envoy\") pod \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " Oct 30 18:25:20.119713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119666 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istiod-ca-cert\") pod \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " Oct 30 18:25:20.119816 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119751 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-socket\") pod \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\" (UID: \"1a4d1c63-04da-46e0-953e-40e5ef9663e1\") " Oct 30 18:25:20.119877 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.119843 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-certs" pod "1a4d1c63-04da-46e0-953e-40e5ef9663e1" (UID: "1a4d1c63-04da-46e0-953e-40e5ef9663e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:20.120046 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.120020 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-credential-socket" pod "1a4d1c63-04da-46e0-953e-40e5ef9663e1" (UID: "1a4d1c63-04da-46e0-953e-40e5ef9663e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:20.120339 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.120159 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-data" pod "1a4d1c63-04da-46e0-953e-40e5ef9663e1" (UID: "1a4d1c63-04da-46e0-953e-40e5ef9663e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:20.120610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.120562 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-socket" pod "1a4d1c63-04da-46e0-953e-40e5ef9663e1" (UID: "1a4d1c63-04da-46e0-953e-40e5ef9663e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:20.120610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.120574 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:20.120610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.120577 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istiod-ca-cert" pod "1a4d1c63-04da-46e0-953e-40e5ef9663e1" (UID: "1a4d1c63-04da-46e0-953e-40e5ef9663e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:20.120610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.120595 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:20.120610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.120609 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:20.121783 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.121763 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-token" pod "1a4d1c63-04da-46e0-953e-40e5ef9663e1" (UID: "1a4d1c63-04da-46e0-953e-40e5ef9663e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:20.121880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.121849 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-kube-api-access-nx7mz" pod "1a4d1c63-04da-46e0-953e-40e5ef9663e1" (UID: "1a4d1c63-04da-46e0-953e-40e5ef9663e1"). InnerVolumeSpecName "kube-api-access-nx7mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:20.121951 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.121933 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-podinfo" pod "1a4d1c63-04da-46e0-953e-40e5ef9663e1" (UID: "1a4d1c63-04da-46e0-953e-40e5ef9663e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:20.122255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.122238 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-envoy" pod "1a4d1c63-04da-46e0-953e-40e5ef9663e1" (UID: "1a4d1c63-04da-46e0-953e-40e5ef9663e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:20.221631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.221569 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:20.221631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.221591 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:20.221631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.221600 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4d1c63-04da-46e0-953e-40e5ef9663e1-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:20.221631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.221608 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:20.221631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.221618 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx7mz\" (UniqueName: \"kubernetes.io/projected/1a4d1c63-04da-46e0-953e-40e5ef9663e1-kube-api-access-nx7mz\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:20.221631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.221626 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4d1c63-04da-46e0-953e-40e5ef9663e1-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:20.445035 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.445010 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj"] Oct 30 18:25:20.456467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:20.456447 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-011d6f-openshift-default-555c887c5f-jkbfj"] Oct 30 18:25:21.063542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:21.063513 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" path="/var/lib/kubelet/pods/1a4d1c63-04da-46e0-953e-40e5ef9663e1/volumes" Oct 30 18:25:38.234947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:38.234914 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m"] Oct 30 18:25:38.235448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:38.235167 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" containerID="cri-o://bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231" gracePeriod=30 Oct 30 18:25:38.254443 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:38.250915 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v"] Oct 30 18:25:38.254443 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:38.251139 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" containerID="cri-o://b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0" gracePeriod=30 Oct 30 18:25:38.283635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:38.283605 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2"] Oct 30 18:25:38.283800 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:38.283781 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" containerID="cri-o://7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3" gracePeriod=30 Oct 30 18:25:41.312502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:41.312467 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:25:41.312961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:41.312524 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:25:43.781320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.781255 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:25:43.800835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.800816 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:25:43.804773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.804754 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:25:43.947989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.947886 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-token\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-token\") pod \"6ca9be35-a071-480e-97bb-68eedec2f8c6\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " Oct 30 18:25:43.948163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.947987 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-data\") pod \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " Oct 30 18:25:43.948163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948019 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-data\") pod \"6ca9be35-a071-480e-97bb-68eedec2f8c6\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " Oct 30 18:25:43.948163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948083 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-certs\") pod \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " Oct 30 18:25:43.948163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948117 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-kube-api-access-wtg96\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-kube-api-access-wtg96\") pod \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " Oct 30 18:25:43.948484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948181 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-credential-socket\") pod \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " Oct 30 18:25:43.948484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948233 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-certs\") pod \"6ca9be35-a071-480e-97bb-68eedec2f8c6\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " Oct 30 18:25:43.948484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948305 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-kube-api-access-dkpvn\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-kube-api-access-dkpvn\") pod \"6ca9be35-a071-480e-97bb-68eedec2f8c6\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " Oct 30 18:25:43.948484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948381 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-socket\") pod \"6ca9be35-a071-480e-97bb-68eedec2f8c6\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " Oct 30 18:25:43.948484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948443 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-certs\") pod \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948523 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-podinfo\") pod \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948561 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-envoy\") pod \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948610 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-token\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-token\") pod \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948698 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istiod-ca-cert\") pod \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948717 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-credential-socket" pod "1c1735e3-e2ea-4e64-a372-e88738de6b4a" (UID: "1c1735e3-e2ea-4e64-a372-e88738de6b4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948775 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-credential-socket\") pod \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948806 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-envoy\") pod \"6ca9be35-a071-480e-97bb-68eedec2f8c6\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948855 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-kube-api-access-nvzzr\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-kube-api-access-nvzzr\") pod \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948927 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-socket\") pod \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\" (UID: \"09ae0d79-41b8-4c59-bcb4-225820e3b21d\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948957 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-data" pod "6ca9be35-a071-480e-97bb-68eedec2f8c6" (UID: "6ca9be35-a071-480e-97bb-68eedec2f8c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.948966 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istiod-ca-cert\") pod \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949019 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-envoy\") pod \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " Oct 30 18:25:43.949103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949072 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-podinfo\") pod \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " Oct 30 18:25:43.949801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949126 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca9be35-a071-480e-97bb-68eedec2f8c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca9be35-a071-480e-97bb-68eedec2f8c6-istiod-ca-cert\") pod \"6ca9be35-a071-480e-97bb-68eedec2f8c6\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " Oct 30 18:25:43.949801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949147 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-certs" pod "09ae0d79-41b8-4c59-bcb4-225820e3b21d" (UID: "09ae0d79-41b8-4c59-bcb4-225820e3b21d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.949801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949191 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-data\") pod \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " Oct 30 18:25:43.949801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949257 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-token\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-token\") pod \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " Oct 30 18:25:43.949801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949439 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-podinfo\") pod \"6ca9be35-a071-480e-97bb-68eedec2f8c6\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " Oct 30 18:25:43.949801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949550 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-socket\") pod \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\" (UID: \"1c1735e3-e2ea-4e64-a372-e88738de6b4a\") " Oct 30 18:25:43.949801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949591 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-credential-socket\") pod \"6ca9be35-a071-480e-97bb-68eedec2f8c6\" (UID: \"6ca9be35-a071-480e-97bb-68eedec2f8c6\") " Oct 30 18:25:43.949801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.949656 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-data" pod "09ae0d79-41b8-4c59-bcb4-225820e3b21d" (UID: "09ae0d79-41b8-4c59-bcb4-225820e3b21d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.951356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.950463 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.951356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.950489 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.951356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.950521 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.951356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.950536 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:43.951356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.951022 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-socket" pod "6ca9be35-a071-480e-97bb-68eedec2f8c6" (UID: "6ca9be35-a071-480e-97bb-68eedec2f8c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.952375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.951904 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-certs" pod "6ca9be35-a071-480e-97bb-68eedec2f8c6" (UID: "6ca9be35-a071-480e-97bb-68eedec2f8c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.952375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.952109 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-certs" pod "1c1735e3-e2ea-4e64-a372-e88738de6b4a" (UID: "1c1735e3-e2ea-4e64-a372-e88738de6b4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.953286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.952738 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-credential-socket" pod "6ca9be35-a071-480e-97bb-68eedec2f8c6" (UID: "6ca9be35-a071-480e-97bb-68eedec2f8c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.960672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.959447 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-socket" pod "09ae0d79-41b8-4c59-bcb4-225820e3b21d" (UID: "09ae0d79-41b8-4c59-bcb4-225820e3b21d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.960672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.959454 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-socket" pod "1c1735e3-e2ea-4e64-a372-e88738de6b4a" (UID: "1c1735e3-e2ea-4e64-a372-e88738de6b4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.960672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.959514 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca9be35-a071-480e-97bb-68eedec2f8c6-istiod-ca-cert" pod "6ca9be35-a071-480e-97bb-68eedec2f8c6" (UID: "6ca9be35-a071-480e-97bb-68eedec2f8c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:43.960672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.959660 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-data" pod "1c1735e3-e2ea-4e64-a372-e88738de6b4a" (UID: "1c1735e3-e2ea-4e64-a372-e88738de6b4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.960672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.959859 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istiod-ca-cert" pod "1c1735e3-e2ea-4e64-a372-e88738de6b4a" (UID: "1c1735e3-e2ea-4e64-a372-e88738de6b4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:43.961134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.961034 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-credential-socket" pod "09ae0d79-41b8-4c59-bcb4-225820e3b21d" (UID: "09ae0d79-41b8-4c59-bcb4-225820e3b21d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.961485 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.961443 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istiod-ca-cert" pod "09ae0d79-41b8-4c59-bcb4-225820e3b21d" (UID: "09ae0d79-41b8-4c59-bcb4-225820e3b21d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:43.961931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.961776 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-kube-api-access-wtg96" pod "1c1735e3-e2ea-4e64-a372-e88738de6b4a" (UID: "1c1735e3-e2ea-4e64-a372-e88738de6b4a"). InnerVolumeSpecName "kube-api-access-wtg96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.961931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.961774 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-podinfo" pod "6ca9be35-a071-480e-97bb-68eedec2f8c6" (UID: "6ca9be35-a071-480e-97bb-68eedec2f8c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:43.961931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.961817 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-envoy" pod "6ca9be35-a071-480e-97bb-68eedec2f8c6" (UID: "6ca9be35-a071-480e-97bb-68eedec2f8c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.961931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.961894 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-kube-api-access-dkpvn" pod "6ca9be35-a071-480e-97bb-68eedec2f8c6" (UID: "6ca9be35-a071-480e-97bb-68eedec2f8c6"). InnerVolumeSpecName "kube-api-access-dkpvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.962412 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.962243 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-token" pod "6ca9be35-a071-480e-97bb-68eedec2f8c6" (UID: "6ca9be35-a071-480e-97bb-68eedec2f8c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.962412 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.962372 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-token" pod "1c1735e3-e2ea-4e64-a372-e88738de6b4a" (UID: "1c1735e3-e2ea-4e64-a372-e88738de6b4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.962573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.962456 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-kube-api-access-nvzzr" pod "09ae0d79-41b8-4c59-bcb4-225820e3b21d" (UID: "09ae0d79-41b8-4c59-bcb4-225820e3b21d"). InnerVolumeSpecName "kube-api-access-nvzzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.962875 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.962841 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-token" pod "09ae0d79-41b8-4c59-bcb4-225820e3b21d" (UID: "09ae0d79-41b8-4c59-bcb4-225820e3b21d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:43.963335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.963308 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-podinfo" pod "1c1735e3-e2ea-4e64-a372-e88738de6b4a" (UID: "1c1735e3-e2ea-4e64-a372-e88738de6b4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:43.963419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.963386 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-envoy" pod "1c1735e3-e2ea-4e64-a372-e88738de6b4a" (UID: "1c1735e3-e2ea-4e64-a372-e88738de6b4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:43.963688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.963660 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-podinfo" pod "09ae0d79-41b8-4c59-bcb4-225820e3b21d" (UID: "09ae0d79-41b8-4c59-bcb4-225820e3b21d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:43.964684 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:43.964647 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-envoy" pod "09ae0d79-41b8-4c59-bcb4-225820e3b21d" (UID: "09ae0d79-41b8-4c59-bcb4-225820e3b21d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:44.052383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052347 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052372 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052388 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052412 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052423 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtg96\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-kube-api-access-wtg96\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052431 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052441 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkpvn\" (UniqueName: \"kubernetes.io/projected/6ca9be35-a071-480e-97bb-68eedec2f8c6-kube-api-access-dkpvn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052451 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052465 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052474 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052481 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052491 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052497 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ae0d79-41b8-4c59-bcb4-225820e3b21d-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052505 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052513 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca9be35-a071-480e-97bb-68eedec2f8c6-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052522 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvzzr\" (UniqueName: \"kubernetes.io/projected/09ae0d79-41b8-4c59-bcb4-225820e3b21d-kube-api-access-nvzzr\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052534 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ae0d79-41b8-4c59-bcb4-225820e3b21d-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052546 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052554 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052561 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052569 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca9be35-a071-480e-97bb-68eedec2f8c6-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052577 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.052593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.052583 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1735e3-e2ea-4e64-a372-e88738de6b4a-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:25:44.280404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.280370 4862 generic.go:356] "Generic (PLEG): container finished" podID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerID="7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3" exitCode=0 Oct 30 18:25:44.280565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.280427 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" Oct 30 18:25:44.280565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.280451 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" event={"ID":"6ca9be35-a071-480e-97bb-68eedec2f8c6","Type":"ContainerDied","Data":"7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3"} Oct 30 18:25:44.280565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.280507 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2" event={"ID":"6ca9be35-a071-480e-97bb-68eedec2f8c6","Type":"ContainerDied","Data":"08b4a6bea3c9fd75e4e8de57d0b5d98fae0724036b422c15d364c77ea4a33f77"} Oct 30 18:25:44.280565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.280527 4862 scope.go:122] "RemoveContainer" containerID="7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3" Oct 30 18:25:44.282291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.282253 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" Oct 30 18:25:44.282411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.282293 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" event={"ID":"09ae0d79-41b8-4c59-bcb4-225820e3b21d","Type":"ContainerDied","Data":"b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0"} Oct 30 18:25:44.282411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.282231 4862 generic.go:356] "Generic (PLEG): container finished" podID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerID="b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0" exitCode=0 Oct 30 18:25:44.282508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.282488 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v" event={"ID":"09ae0d79-41b8-4c59-bcb4-225820e3b21d","Type":"ContainerDied","Data":"335b39e8fa62a6be60add43332faf03b1a955266bac4dad8466e93524d64c0b0"} Oct 30 18:25:44.287300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.284447 4862 generic.go:356] "Generic (PLEG): container finished" podID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerID="bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231" exitCode=0 Oct 30 18:25:44.287300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.284502 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" Oct 30 18:25:44.287300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.284533 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" event={"ID":"1c1735e3-e2ea-4e64-a372-e88738de6b4a","Type":"ContainerDied","Data":"bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231"} Oct 30 18:25:44.287300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.284575 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m" event={"ID":"1c1735e3-e2ea-4e64-a372-e88738de6b4a","Type":"ContainerDied","Data":"698feb0f1119344cdad2196e5d0704364ba218fa886d483a0f9c1296f2df5f26"} Oct 30 18:25:44.292681 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.292666 4862 scope.go:122] "RemoveContainer" containerID="7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde" Oct 30 18:25:44.304395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.304375 4862 scope.go:122] "RemoveContainer" containerID="7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3" Oct 30 18:25:44.304687 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:25:44.304668 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3\": container with ID starting with 7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3 not found: ID does not exist" containerID="7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3" Oct 30 18:25:44.304774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.304697 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3"} err="failed to get container status \"7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3\": rpc error: code = NotFound desc = could not find container \"7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3\": container with ID starting with 7712a3d8e369cda2a23e6d7161486d24430146bf5803c4b90f9cc8fc941828c3 not found: ID does not exist" Oct 30 18:25:44.304774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.304718 4862 scope.go:122] "RemoveContainer" containerID="7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde" Oct 30 18:25:44.305007 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:25:44.304986 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde\": container with ID starting with 7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde not found: ID does not exist" containerID="7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde" Oct 30 18:25:44.305075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.305014 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde"} err="failed to get container status \"7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde\": rpc error: code = NotFound desc = could not find container \"7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde\": container with ID starting with 7eb062866d22fb4a329f324af835af84c3988bbb88b76a5c30ad6827760e4dde not found: ID does not exist" Oct 30 18:25:44.305075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.305033 4862 scope.go:122] "RemoveContainer" containerID="b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0" Oct 30 18:25:44.313558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.313543 4862 scope.go:122] "RemoveContainer" containerID="b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0" Oct 30 18:25:44.313795 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:25:44.313778 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0\": container with ID starting with b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0 not found: ID does not exist" containerID="b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0" Oct 30 18:25:44.313866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.313799 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0"} err="failed to get container status \"b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0\": rpc error: code = NotFound desc = could not find container \"b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0\": container with ID starting with b32b4df522f0593d7b9e6ba8e97ecd99ee1c743a7d12a55f82580fe24abb84d0 not found: ID does not exist" Oct 30 18:25:44.313866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.313813 4862 scope.go:122] "RemoveContainer" containerID="bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231" Oct 30 18:25:44.322159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.322140 4862 scope.go:122] "RemoveContainer" containerID="d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25" Oct 30 18:25:44.330496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.330479 4862 scope.go:122] "RemoveContainer" containerID="bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231" Oct 30 18:25:44.330775 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:25:44.330758 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231\": container with ID starting with bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231 not found: ID does not exist" containerID="bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231" Oct 30 18:25:44.330860 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.330782 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231"} err="failed to get container status \"bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231\": rpc error: code = NotFound desc = could not find container \"bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231\": container with ID starting with bc6005ce7742ecaf0556617217ba1a1ef0da8ba96f2662cf489369d462d38231 not found: ID does not exist" Oct 30 18:25:44.330860 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.330798 4862 scope.go:122] "RemoveContainer" containerID="d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25" Oct 30 18:25:44.331067 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:25:44.331049 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25\": container with ID starting with d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25 not found: ID does not exist" containerID="d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25" Oct 30 18:25:44.331121 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.331074 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25"} err="failed to get container status \"d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25\": rpc error: code = NotFound desc = could not find container \"d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25\": container with ID starting with d6c33f9d83e421a5ed38dd5ad21d6763f2e79ad72cc876eaa77222e2cfde0a25 not found: ID does not exist" Oct 30 18:25:44.349205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.349179 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2"] Oct 30 18:25:44.360673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.360649 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-mpbz2"] Oct 30 18:25:44.407402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.407349 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v"] Oct 30 18:25:44.410445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.410419 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-xhh6v"] Oct 30 18:25:44.447403 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.447376 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m"] Oct 30 18:25:44.455720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:44.455696 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-smn7m"] Oct 30 18:25:45.062950 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:45.062912 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" path="/var/lib/kubelet/pods/09ae0d79-41b8-4c59-bcb4-225820e3b21d/volumes" Oct 30 18:25:45.063464 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:45.063442 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" path="/var/lib/kubelet/pods/1c1735e3-e2ea-4e64-a372-e88738de6b4a/volumes" Oct 30 18:25:45.063885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:45.063871 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" path="/var/lib/kubelet/pods/6ca9be35-a071-480e-97bb-68eedec2f8c6/volumes" Oct 30 18:25:47.445613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:25:47.445571 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:25:49.869749 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:25:49.869711 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:26:08.180861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:08.180790 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg"] Oct 30 18:26:08.181245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:08.180987 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" containerID="cri-o://b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc" gracePeriod=30 Oct 30 18:26:13.492116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.492086 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:26:13.507752 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.507724 4862 generic.go:356] "Generic (PLEG): container finished" podID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerID="b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc" exitCode=0 Oct 30 18:26:13.507892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.507771 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" event={"ID":"2d2613d9-58e1-438d-b256-d6bc303cd461","Type":"ContainerDied","Data":"b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc"} Oct 30 18:26:13.507892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.507783 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" Oct 30 18:26:13.507892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.507828 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg" event={"ID":"2d2613d9-58e1-438d-b256-d6bc303cd461","Type":"ContainerDied","Data":"8e087971f5ac8a0382b7553a4d48f949cfa7e6b5a178c3b5b25eeb4c980fc407"} Oct 30 18:26:13.507892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.507859 4862 scope.go:122] "RemoveContainer" containerID="b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc" Oct 30 18:26:13.520006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.519978 4862 scope.go:122] "RemoveContainer" containerID="b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc" Oct 30 18:26:13.520329 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:26:13.520303 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc\": container with ID starting with b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc not found: ID does not exist" containerID="b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc" Oct 30 18:26:13.520437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.520341 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc"} err="failed to get container status \"b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc\": rpc error: code = NotFound desc = could not find container \"b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc\": container with ID starting with b66816a30b42c944886b48e7ede8b4c4af0c7d6767f8b96b71332c5eeb9cacbc not found: ID does not exist" Oct 30 18:26:13.575871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.575841 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-socket\") pod \"2d2613d9-58e1-438d-b256-d6bc303cd461\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " Oct 30 18:26:13.576046 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.575927 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-kube-api-access-gclrq\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-kube-api-access-gclrq\") pod \"2d2613d9-58e1-438d-b256-d6bc303cd461\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " Oct 30 18:26:13.576046 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.575989 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-certs\") pod \"2d2613d9-58e1-438d-b256-d6bc303cd461\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " Oct 30 18:26:13.576046 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576030 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2613d9-58e1-438d-b256-d6bc303cd461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2613d9-58e1-438d-b256-d6bc303cd461-istiod-ca-cert\") pod \"2d2613d9-58e1-438d-b256-d6bc303cd461\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " Oct 30 18:26:13.576234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576083 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-credential-socket\") pod \"2d2613d9-58e1-438d-b256-d6bc303cd461\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " Oct 30 18:26:13.576234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576113 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-envoy\") pod \"2d2613d9-58e1-438d-b256-d6bc303cd461\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " Oct 30 18:26:13.576234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576128 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-socket" pod "2d2613d9-58e1-438d-b256-d6bc303cd461" (UID: "2d2613d9-58e1-438d-b256-d6bc303cd461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:13.576234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576171 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-data\") pod \"2d2613d9-58e1-438d-b256-d6bc303cd461\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " Oct 30 18:26:13.576234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576230 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-token\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-token\") pod \"2d2613d9-58e1-438d-b256-d6bc303cd461\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " Oct 30 18:26:13.576557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576301 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-podinfo\") pod \"2d2613d9-58e1-438d-b256-d6bc303cd461\" (UID: \"2d2613d9-58e1-438d-b256-d6bc303cd461\") " Oct 30 18:26:13.576557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576354 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-certs" pod "2d2613d9-58e1-438d-b256-d6bc303cd461" (UID: "2d2613d9-58e1-438d-b256-d6bc303cd461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:13.577458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576934 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.577458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.576960 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.577458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.577209 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-credential-socket" pod "2d2613d9-58e1-438d-b256-d6bc303cd461" (UID: "2d2613d9-58e1-438d-b256-d6bc303cd461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:13.577458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.577426 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2613d9-58e1-438d-b256-d6bc303cd461-istiod-ca-cert" pod "2d2613d9-58e1-438d-b256-d6bc303cd461" (UID: "2d2613d9-58e1-438d-b256-d6bc303cd461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:13.578073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.578054 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-data" pod "2d2613d9-58e1-438d-b256-d6bc303cd461" (UID: "2d2613d9-58e1-438d-b256-d6bc303cd461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:13.579395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.579351 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-kube-api-access-gclrq" pod "2d2613d9-58e1-438d-b256-d6bc303cd461" (UID: "2d2613d9-58e1-438d-b256-d6bc303cd461"). InnerVolumeSpecName "kube-api-access-gclrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:13.579628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.579607 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-podinfo" pod "2d2613d9-58e1-438d-b256-d6bc303cd461" (UID: "2d2613d9-58e1-438d-b256-d6bc303cd461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:13.580626 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.580604 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-token" pod "2d2613d9-58e1-438d-b256-d6bc303cd461" (UID: "2d2613d9-58e1-438d-b256-d6bc303cd461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:13.580767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.580744 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-envoy" pod "2d2613d9-58e1-438d-b256-d6bc303cd461" (UID: "2d2613d9-58e1-438d-b256-d6bc303cd461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:13.678630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.678599 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gclrq\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-kube-api-access-gclrq\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.678775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.678634 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2613d9-58e1-438d-b256-d6bc303cd461-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.678775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.678649 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.678775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.678663 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.678775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.678676 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.678775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.678688 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.678775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.678701 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2613d9-58e1-438d-b256-d6bc303cd461-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:13.858656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.858623 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg"] Oct 30 18:26:13.870660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:13.870633 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-74gqg"] Oct 30 18:26:15.062780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:15.062750 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" path="/var/lib/kubelet/pods/2d2613d9-58e1-438d-b256-d6bc303cd461/volumes" Oct 30 18:26:19.871399 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:26:19.871363 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:26:23.214992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:23.214956 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc"] Oct 30 18:26:23.215531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:23.215146 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" containerID="cri-o://9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee" gracePeriod=30 Oct 30 18:26:23.226710 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:23.226674 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6"] Oct 30 18:26:23.226894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:23.226874 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" containerID="cri-o://12f59539367b860ab9c9ebadae64280523069b75d7d8d8079a90ba5f4b849339" gracePeriod=30 Oct 30 18:26:23.307192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:23.307153 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p"] Oct 30 18:26:23.308505 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:23.308472 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" containerID="cri-o://923cf4d0528d3866e750dee6e85d9f4c15679b29acd2e45a454b214d91d9af24" gracePeriod=30 Oct 30 18:26:28.409567 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.404171 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:26:28.411649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.411615 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 18:26:28.411787 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.411670 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 18:26:28.587432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.587112 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:26:28.646312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.644141 4862 generic.go:356] "Generic (PLEG): container finished" podID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerID="923cf4d0528d3866e750dee6e85d9f4c15679b29acd2e45a454b214d91d9af24" exitCode=0 Oct 30 18:26:28.646312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.644351 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" event={"ID":"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa","Type":"ContainerDied","Data":"923cf4d0528d3866e750dee6e85d9f4c15679b29acd2e45a454b214d91d9af24"} Oct 30 18:26:28.646719 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.646693 4862 generic.go:356] "Generic (PLEG): container finished" podID="47113181-eed4-4806-afa0-a05250b7b7a8" containerID="9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee" exitCode=0 Oct 30 18:26:28.646849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.646778 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" event={"ID":"47113181-eed4-4806-afa0-a05250b7b7a8","Type":"ContainerDied","Data":"9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee"} Oct 30 18:26:28.646906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.646853 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" event={"ID":"47113181-eed4-4806-afa0-a05250b7b7a8","Type":"ContainerDied","Data":"37a88dcaf3a32bb4b25024894b40b40c74386e4a02a71b85feface4fc53d9fd8"} Oct 30 18:26:28.646906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.646885 4862 scope.go:122] "RemoveContainer" containerID="9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee" Oct 30 18:26:28.647060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.647045 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc" Oct 30 18:26:28.650964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.650935 4862 generic.go:356] "Generic (PLEG): container finished" podID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerID="12f59539367b860ab9c9ebadae64280523069b75d7d8d8079a90ba5f4b849339" exitCode=0 Oct 30 18:26:28.651084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.651009 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" event={"ID":"ef8b5882-396e-446e-9efc-b7c11e0d454a","Type":"ContainerDied","Data":"12f59539367b860ab9c9ebadae64280523069b75d7d8d8079a90ba5f4b849339"} Oct 30 18:26:28.659028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.659005 4862 scope.go:122] "RemoveContainer" containerID="9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee" Oct 30 18:26:28.659428 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:26:28.659405 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee\": container with ID starting with 9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee not found: ID does not exist" containerID="9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee" Oct 30 18:26:28.659515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.659442 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee"} err="failed to get container status \"9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee\": rpc error: code = NotFound desc = could not find container \"9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee\": container with ID starting with 9be09833314b56e1a9c781a97fdcd72cffc141495ac21651306ca4917c81a0ee not found: ID does not exist" Oct 30 18:26:28.691812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.691790 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:26:28.751638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.751612 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:26:28.774373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774340 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-socket\") pod \"47113181-eed4-4806-afa0-a05250b7b7a8\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " Oct 30 18:26:28.774544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774524 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-envoy\") pod \"47113181-eed4-4806-afa0-a05250b7b7a8\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " Oct 30 18:26:28.774627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774560 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-kube-api-access-mlqjr\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-kube-api-access-mlqjr\") pod \"47113181-eed4-4806-afa0-a05250b7b7a8\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " Oct 30 18:26:28.774627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774568 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-socket" pod "47113181-eed4-4806-afa0-a05250b7b7a8" (UID: "47113181-eed4-4806-afa0-a05250b7b7a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.774627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774602 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-data\") pod \"47113181-eed4-4806-afa0-a05250b7b7a8\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " Oct 30 18:26:28.774794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774648 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-credential-socket\") pod \"47113181-eed4-4806-afa0-a05250b7b7a8\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " Oct 30 18:26:28.774794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774709 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47113181-eed4-4806-afa0-a05250b7b7a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47113181-eed4-4806-afa0-a05250b7b7a8-istiod-ca-cert\") pod \"47113181-eed4-4806-afa0-a05250b7b7a8\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " Oct 30 18:26:28.774794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774761 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47113181-eed4-4806-afa0-a05250b7b7a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47113181-eed4-4806-afa0-a05250b7b7a8-istio-podinfo\") pod \"47113181-eed4-4806-afa0-a05250b7b7a8\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " Oct 30 18:26:28.774794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774788 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-certs\") pod \"47113181-eed4-4806-afa0-a05250b7b7a8\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " Oct 30 18:26:28.775009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.774813 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-istio-token\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-istio-token\") pod \"47113181-eed4-4806-afa0-a05250b7b7a8\" (UID: \"47113181-eed4-4806-afa0-a05250b7b7a8\") " Oct 30 18:26:28.775413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.775280 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47113181-eed4-4806-afa0-a05250b7b7a8-istiod-ca-cert" pod "47113181-eed4-4806-afa0-a05250b7b7a8" (UID: "47113181-eed4-4806-afa0-a05250b7b7a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:28.775541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.775385 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.775541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.775510 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-credential-socket" pod "47113181-eed4-4806-afa0-a05250b7b7a8" (UID: "47113181-eed4-4806-afa0-a05250b7b7a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.775906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.775876 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-certs" pod "47113181-eed4-4806-afa0-a05250b7b7a8" (UID: "47113181-eed4-4806-afa0-a05250b7b7a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.776404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.776380 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-data" pod "47113181-eed4-4806-afa0-a05250b7b7a8" (UID: "47113181-eed4-4806-afa0-a05250b7b7a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.777204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.777177 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-kube-api-access-mlqjr" pod "47113181-eed4-4806-afa0-a05250b7b7a8" (UID: "47113181-eed4-4806-afa0-a05250b7b7a8"). InnerVolumeSpecName "kube-api-access-mlqjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.777527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.777509 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47113181-eed4-4806-afa0-a05250b7b7a8-istio-podinfo" pod "47113181-eed4-4806-afa0-a05250b7b7a8" (UID: "47113181-eed4-4806-afa0-a05250b7b7a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:28.777822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.777800 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-istio-token" pod "47113181-eed4-4806-afa0-a05250b7b7a8" (UID: "47113181-eed4-4806-afa0-a05250b7b7a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.778425 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.778409 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-envoy" pod "47113181-eed4-4806-afa0-a05250b7b7a8" (UID: "47113181-eed4-4806-afa0-a05250b7b7a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.876518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876483 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istiod-ca-cert\") pod \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " Oct 30 18:26:28.876717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876543 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-kube-api-access-z8fzt\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-kube-api-access-z8fzt\") pod \"ef8b5882-396e-446e-9efc-b7c11e0d454a\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " Oct 30 18:26:28.876717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876576 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-data\") pod \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " Oct 30 18:26:28.876717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876648 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-socket\") pod \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " Oct 30 18:26:28.876907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876724 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8b5882-396e-446e-9efc-b7c11e0d454a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b5882-396e-446e-9efc-b7c11e0d454a-istiod-ca-cert\") pod \"ef8b5882-396e-446e-9efc-b7c11e0d454a\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " Oct 30 18:26:28.876907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876765 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-podinfo\") pod \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " Oct 30 18:26:28.876907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876808 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-token\") pod \"ef8b5882-396e-446e-9efc-b7c11e0d454a\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " Oct 30 18:26:28.876907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876857 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-socket\") pod \"ef8b5882-396e-446e-9efc-b7c11e0d454a\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " Oct 30 18:26:28.876907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876885 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-podinfo\") pod \"ef8b5882-396e-446e-9efc-b7c11e0d454a\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " Oct 30 18:26:28.877169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876921 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-data\") pod \"ef8b5882-396e-446e-9efc-b7c11e0d454a\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " Oct 30 18:26:28.877169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876922 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istiod-ca-cert" pod "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" (UID: "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:28.877169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876951 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-data" pod "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" (UID: "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.877169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.876994 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-certs\") pod \"ef8b5882-396e-446e-9efc-b7c11e0d454a\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " Oct 30 18:26:28.877169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877025 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-envoy\") pod \"ef8b5882-396e-446e-9efc-b7c11e0d454a\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " Oct 30 18:26:28.877169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877053 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-credential-socket\") pod \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " Oct 30 18:26:28.877169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877096 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-credential-socket\") pod \"ef8b5882-396e-446e-9efc-b7c11e0d454a\" (UID: \"ef8b5882-396e-446e-9efc-b7c11e0d454a\") " Oct 30 18:26:28.877169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877130 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-kube-api-access-xhf2q\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-kube-api-access-xhf2q\") pod \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " Oct 30 18:26:28.877169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877156 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-token\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-token\") pod \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " Oct 30 18:26:28.877733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877218 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-certs\") pod \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " Oct 30 18:26:28.877733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877217 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-socket" pod "ef8b5882-396e-446e-9efc-b7c11e0d454a" (UID: "ef8b5882-396e-446e-9efc-b7c11e0d454a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.877733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877298 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-envoy\") pod \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\" (UID: \"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa\") " Oct 30 18:26:28.877733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877585 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-certs" pod "ef8b5882-396e-446e-9efc-b7c11e0d454a" (UID: "ef8b5882-396e-446e-9efc-b7c11e0d454a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.877733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.877596 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-socket" pod "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" (UID: "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.878042 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878016 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47113181-eed4-4806-afa0-a05250b7b7a8-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878042 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878037 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878077 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47113181-eed4-4806-afa0-a05250b7b7a8-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878094 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878107 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878120 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878132 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878162 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878175 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlqjr\" (UniqueName: \"kubernetes.io/projected/47113181-eed4-4806-afa0-a05250b7b7a8-kube-api-access-mlqjr\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878188 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878200 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878211 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878240 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47113181-eed4-4806-afa0-a05250b7b7a8-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.878823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878795 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-certs" pod "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" (UID: "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.878823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.878811 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-credential-socket" pod "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" (UID: "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.879166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.879041 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8b5882-396e-446e-9efc-b7c11e0d454a-istiod-ca-cert" pod "ef8b5882-396e-446e-9efc-b7c11e0d454a" (UID: "ef8b5882-396e-446e-9efc-b7c11e0d454a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:28.879166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.879095 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-credential-socket" pod "ef8b5882-396e-446e-9efc-b7c11e0d454a" (UID: "ef8b5882-396e-446e-9efc-b7c11e0d454a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.879166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.879145 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-data" pod "ef8b5882-396e-446e-9efc-b7c11e0d454a" (UID: "ef8b5882-396e-446e-9efc-b7c11e0d454a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.882022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.881983 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-envoy" pod "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" (UID: "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.882276 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.882236 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-token" pod "ef8b5882-396e-446e-9efc-b7c11e0d454a" (UID: "ef8b5882-396e-446e-9efc-b7c11e0d454a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.882409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.882395 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-envoy" pod "ef8b5882-396e-446e-9efc-b7c11e0d454a" (UID: "ef8b5882-396e-446e-9efc-b7c11e0d454a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:28.882480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.882470 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-podinfo" pod "ef8b5882-396e-446e-9efc-b7c11e0d454a" (UID: "ef8b5882-396e-446e-9efc-b7c11e0d454a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:28.882541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.882522 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-podinfo" pod "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" (UID: "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:28.882634 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.882599 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-token" pod "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" (UID: "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.882714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.882660 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-kube-api-access-z8fzt" pod "ef8b5882-396e-446e-9efc-b7c11e0d454a" (UID: "ef8b5882-396e-446e-9efc-b7c11e0d454a"). InnerVolumeSpecName "kube-api-access-z8fzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.883401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.883382 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-kube-api-access-xhf2q" pod "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" (UID: "a8c89f09-f8de-4a56-9a63-e30e1e6f27aa"). InnerVolumeSpecName "kube-api-access-xhf2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:28.978994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.978967 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8b5882-396e-446e-9efc-b7c11e0d454a-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.978994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.978993 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.978994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979002 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979011 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979019 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979028 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979035 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979043 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8b5882-396e-446e-9efc-b7c11e0d454a-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979051 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhf2q\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-kube-api-access-xhf2q\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979060 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979067 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979075 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:28.979228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.979082 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8fzt\" (UniqueName: \"kubernetes.io/projected/ef8b5882-396e-446e-9efc-b7c11e0d454a-kube-api-access-z8fzt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.000012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:28.999979 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc"] Oct 30 18:26:29.004687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.004663 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-tzhkc"] Oct 30 18:26:29.062641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.062613 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" path="/var/lib/kubelet/pods/47113181-eed4-4806-afa0-a05250b7b7a8/volumes" Oct 30 18:26:29.658997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.658965 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" Oct 30 18:26:29.659528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.658962 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p" event={"ID":"a8c89f09-f8de-4a56-9a63-e30e1e6f27aa","Type":"ContainerDied","Data":"4495e3f0659b8b99552d6790795296e1e0e059d830bba5fef7266d7da5a25285"} Oct 30 18:26:29.659528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.659107 4862 scope.go:122] "RemoveContainer" containerID="923cf4d0528d3866e750dee6e85d9f4c15679b29acd2e45a454b214d91d9af24" Oct 30 18:26:29.661931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.661904 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" event={"ID":"ef8b5882-396e-446e-9efc-b7c11e0d454a","Type":"ContainerDied","Data":"84425179de4f93472b8d0418b301fce3cc564d49f0b54fcaf577342725e9176f"} Oct 30 18:26:29.662058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.661969 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6" Oct 30 18:26:29.670169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.670152 4862 scope.go:122] "RemoveContainer" containerID="12f59539367b860ab9c9ebadae64280523069b75d7d8d8079a90ba5f4b849339" Oct 30 18:26:29.724833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.724801 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6"] Oct 30 18:26:29.736639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.736612 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-chrr6"] Oct 30 18:26:29.766928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.766901 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p"] Oct 30 18:26:29.770676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:29.770652 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mjl8p"] Oct 30 18:26:31.063338 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:31.063301 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" path="/var/lib/kubelet/pods/a8c89f09-f8de-4a56-9a63-e30e1e6f27aa/volumes" Oct 30 18:26:31.063790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:31.063776 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" path="/var/lib/kubelet/pods/ef8b5882-396e-446e-9efc-b7c11e0d454a/volumes" Oct 30 18:26:49.887046 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:26:49.887004 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:26:57.598423 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:26:57.598383 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:27:19.869697 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:27:19.869661 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:27:26.165976 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:27:26.165929 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:27:49.869634 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:27:49.869593 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:28:04.051578 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:04.051542 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:28:19.875229 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:28:19.875186 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:28:21.418237 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:21.418209 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:28:21.418827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:21.418809 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:28:27.915000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:27.914959 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8"] Oct 30 18:28:27.915539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:27.915200 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" podUID="b86e3b0b-0fe1-417e-b000-e3f84310f2f8" containerName="maas-api" containerID="cri-o://a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa" gracePeriod=30 Oct 30 18:28:28.422286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.409311 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:28:28.451229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.451136 4862 generic.go:356] "Generic (PLEG): container finished" podID="b86e3b0b-0fe1-417e-b000-e3f84310f2f8" containerID="a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa" exitCode=0 Oct 30 18:28:28.451656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.451631 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" event={"ID":"b86e3b0b-0fe1-417e-b000-e3f84310f2f8","Type":"ContainerDied","Data":"a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa"} Oct 30 18:28:28.451828 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.451812 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" event={"ID":"b86e3b0b-0fe1-417e-b000-e3f84310f2f8","Type":"ContainerDied","Data":"7085a93a4bc616b366ee7ed71185dcf43008f8282386b6cba7e200e1bb9b1cb6"} Oct 30 18:28:28.452524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.451930 4862 scope.go:122] "RemoveContainer" containerID="a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa" Oct 30 18:28:28.452524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.452124 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8" Oct 30 18:28:28.485656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.485630 4862 scope.go:122] "RemoveContainer" containerID="a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa" Oct 30 18:28:28.486097 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:28:28.486056 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa\": container with ID starting with a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa not found: ID does not exist" containerID="a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa" Oct 30 18:28:28.486231 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.486100 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa"} err="failed to get container status \"a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa\": rpc error: code = NotFound desc = could not find container \"a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa\": container with ID starting with a067f0fba729154ef6dfc39df1f4fe5b15e51f2285b67d3335ed664adf4c29aa not found: ID does not exist" Oct 30 18:28:28.599685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.599658 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-metrics-certs\") pod \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " Oct 30 18:28:28.599885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.599715 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-kube-api-access-zjm52\" (UniqueName: \"kubernetes.io/projected/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-kube-api-access-zjm52\") pod \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " Oct 30 18:28:28.599885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.599820 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-openshift-service-ca\") pod \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " Oct 30 18:28:28.599885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.599858 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-maas-api-tls\") pod \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\" (UID: \"b86e3b0b-0fe1-417e-b000-e3f84310f2f8\") " Oct 30 18:28:28.600213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.600186 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-openshift-service-ca" pod "b86e3b0b-0fe1-417e-b000-e3f84310f2f8" (UID: "b86e3b0b-0fe1-417e-b000-e3f84310f2f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:28.600364 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.600326 4862 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-openshift-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.602083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.602057 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-metrics-certs" pod "b86e3b0b-0fe1-417e-b000-e3f84310f2f8" (UID: "b86e3b0b-0fe1-417e-b000-e3f84310f2f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:28.602607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.602584 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-maas-api-tls" pod "b86e3b0b-0fe1-417e-b000-e3f84310f2f8" (UID: "b86e3b0b-0fe1-417e-b000-e3f84310f2f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:28.602706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.602678 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-kube-api-access-zjm52" pod "b86e3b0b-0fe1-417e-b000-e3f84310f2f8" (UID: "b86e3b0b-0fe1-417e-b000-e3f84310f2f8"). InnerVolumeSpecName "kube-api-access-zjm52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:28.701614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.701558 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjm52\" (UniqueName: \"kubernetes.io/projected/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-kube-api-access-zjm52\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.701614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.701580 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-maas-api-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.701614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.701590 4862 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86e3b0b-0fe1-417e-b000-e3f84310f2f8-metrics-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:28.796591 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.796567 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8"] Oct 30 18:28:28.799888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:28.799869 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a4f3f0-566b9fb947-x46z8"] Oct 30 18:28:29.063192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:29.063156 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86e3b0b-0fe1-417e-b000-e3f84310f2f8" path="/var/lib/kubelet/pods/b86e3b0b-0fe1-417e-b000-e3f84310f2f8/volumes" Oct 30 18:28:38.253498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:38.253464 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6"] Oct 30 18:28:38.254100 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:38.253677 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" containerID="cri-o://b3fbfc3714bfc125ad50125030b8476b74aef303e9345a193782afb69b8423b7" gracePeriod=30 Oct 30 18:28:43.552153 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.552123 4862 generic.go:356] "Generic (PLEG): container finished" podID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerID="b3fbfc3714bfc125ad50125030b8476b74aef303e9345a193782afb69b8423b7" exitCode=0 Oct 30 18:28:43.552511 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.552159 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" event={"ID":"42fc7558-2c7b-45af-ad4f-4091b840ee5c","Type":"ContainerDied","Data":"b3fbfc3714bfc125ad50125030b8476b74aef303e9345a193782afb69b8423b7"} Oct 30 18:28:43.552511 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.552211 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" event={"ID":"42fc7558-2c7b-45af-ad4f-4091b840ee5c","Type":"ContainerDied","Data":"e8defdfcf80f1613df9ea11af02f554dfaefc2d95b2e320d096268b8560f9c5b"} Oct 30 18:28:43.552511 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.552228 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8defdfcf80f1613df9ea11af02f554dfaefc2d95b2e320d096268b8560f9c5b" Oct 30 18:28:43.552988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.552972 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:28:43.684498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684468 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istiod-ca-cert\") pod \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " Oct 30 18:28:43.684673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684519 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-podinfo\") pod \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " Oct 30 18:28:43.684673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684628 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-socket\") pod \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " Oct 30 18:28:43.684758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684676 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-certs\") pod \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " Oct 30 18:28:43.684758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684738 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-envoy\") pod \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " Oct 30 18:28:43.684862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684766 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-credential-socket\") pod \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " Oct 30 18:28:43.684862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684792 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-token\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-token\") pod \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " Oct 30 18:28:43.684862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684848 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-kube-api-access-6v6jp\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-kube-api-access-6v6jp\") pod \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " Oct 30 18:28:43.685023 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684901 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-socket" pod "42fc7558-2c7b-45af-ad4f-4091b840ee5c" (UID: "42fc7558-2c7b-45af-ad4f-4091b840ee5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:43.685023 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.684910 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-data\") pod \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\" (UID: \"42fc7558-2c7b-45af-ad4f-4091b840ee5c\") " Oct 30 18:28:43.685023 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.685003 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-certs" pod "42fc7558-2c7b-45af-ad4f-4091b840ee5c" (UID: "42fc7558-2c7b-45af-ad4f-4091b840ee5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:43.685193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.685020 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-credential-socket" pod "42fc7558-2c7b-45af-ad4f-4091b840ee5c" (UID: "42fc7558-2c7b-45af-ad4f-4091b840ee5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:43.685193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.685117 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-data" pod "42fc7558-2c7b-45af-ad4f-4091b840ee5c" (UID: "42fc7558-2c7b-45af-ad4f-4091b840ee5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:43.685353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.685209 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istiod-ca-cert" pod "42fc7558-2c7b-45af-ad4f-4091b840ee5c" (UID: "42fc7558-2c7b-45af-ad4f-4091b840ee5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:43.685543 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.685523 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:43.685799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.685547 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:43.685799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.685561 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:43.685799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.685575 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:43.685799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.685588 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:43.687064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.687042 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-podinfo" pod "42fc7558-2c7b-45af-ad4f-4091b840ee5c" (UID: "42fc7558-2c7b-45af-ad4f-4091b840ee5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:43.687337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.687315 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-kube-api-access-6v6jp" pod "42fc7558-2c7b-45af-ad4f-4091b840ee5c" (UID: "42fc7558-2c7b-45af-ad4f-4091b840ee5c"). InnerVolumeSpecName "kube-api-access-6v6jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:43.687546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.687517 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-envoy" pod "42fc7558-2c7b-45af-ad4f-4091b840ee5c" (UID: "42fc7558-2c7b-45af-ad4f-4091b840ee5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:43.687546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.687527 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-token" pod "42fc7558-2c7b-45af-ad4f-4091b840ee5c" (UID: "42fc7558-2c7b-45af-ad4f-4091b840ee5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:43.787206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.787138 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:43.787206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.787160 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:43.787206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.787171 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v6jp\" (UniqueName: \"kubernetes.io/projected/42fc7558-2c7b-45af-ad4f-4091b840ee5c-kube-api-access-6v6jp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:43.787206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:43.787181 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fc7558-2c7b-45af-ad4f-4091b840ee5c-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:28:44.559471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:44.559441 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6" Oct 30 18:28:44.628056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:44.628034 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6"] Oct 30 18:28:44.640411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:44.640386 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wgrr6"] Oct 30 18:28:45.062449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:45.062420 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" path="/var/lib/kubelet/pods/42fc7558-2c7b-45af-ad4f-4091b840ee5c/volumes" Oct 30 18:28:46.893433 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:28:46.893401 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:28:49.868615 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:28:49.868576 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:29:02.901167 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:02.900951 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls"] Oct 30 18:29:02.901819 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:02.901307 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" containerID="cri-o://1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9" gracePeriod=30 Oct 30 18:29:08.176643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.176583 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:29:08.269039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269008 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-data\") pod \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " Oct 30 18:29:08.269212 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269074 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-token\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-token\") pod \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " Oct 30 18:29:08.269212 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269188 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istiod-ca-cert\") pod \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " Oct 30 18:29:08.269342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269229 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-envoy\") pod \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " Oct 30 18:29:08.269342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269277 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-podinfo\") pod \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " Oct 30 18:29:08.269342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269320 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-data" pod "ba535e0e-ea07-4d37-8c4c-747b060a62dd" (UID: "ba535e0e-ea07-4d37-8c4c-747b060a62dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:08.269473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269329 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-kube-api-access-glcm4\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-kube-api-access-glcm4\") pod \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " Oct 30 18:29:08.269473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269404 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-credential-socket\") pod \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " Oct 30 18:29:08.269558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269534 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-certs\") pod \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " Oct 30 18:29:08.269615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269566 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-socket\") pod \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\" (UID: \"ba535e0e-ea07-4d37-8c4c-747b060a62dd\") " Oct 30 18:29:08.269672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269625 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istiod-ca-cert" pod "ba535e0e-ea07-4d37-8c4c-747b060a62dd" (UID: "ba535e0e-ea07-4d37-8c4c-747b060a62dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:08.269852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269820 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-credential-socket" pod "ba535e0e-ea07-4d37-8c4c-747b060a62dd" (UID: "ba535e0e-ea07-4d37-8c4c-747b060a62dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:08.269945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.269874 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-certs" pod "ba535e0e-ea07-4d37-8c4c-747b060a62dd" (UID: "ba535e0e-ea07-4d37-8c4c-747b060a62dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:08.270049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.270026 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-socket" pod "ba535e0e-ea07-4d37-8c4c-747b060a62dd" (UID: "ba535e0e-ea07-4d37-8c4c-747b060a62dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:08.271134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.270735 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.271134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.270762 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.271134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.270776 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.271134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.270789 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.271134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.270801 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.272033 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.272011 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-kube-api-access-glcm4" pod "ba535e0e-ea07-4d37-8c4c-747b060a62dd" (UID: "ba535e0e-ea07-4d37-8c4c-747b060a62dd"). InnerVolumeSpecName "kube-api-access-glcm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:08.272120 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.272101 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-envoy" pod "ba535e0e-ea07-4d37-8c4c-747b060a62dd" (UID: "ba535e0e-ea07-4d37-8c4c-747b060a62dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:08.272360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.272339 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-podinfo" pod "ba535e0e-ea07-4d37-8c4c-747b060a62dd" (UID: "ba535e0e-ea07-4d37-8c4c-747b060a62dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:08.272428 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.272371 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-token" pod "ba535e0e-ea07-4d37-8c4c-747b060a62dd" (UID: "ba535e0e-ea07-4d37-8c4c-747b060a62dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:08.371972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.371936 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glcm4\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-kube-api-access-glcm4\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.371972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.371971 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.371972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.371980 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.372141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.371988 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba535e0e-ea07-4d37-8c4c-747b060a62dd-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:29:08.714098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.714059 4862 generic.go:356] "Generic (PLEG): container finished" podID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerID="1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9" exitCode=0 Oct 30 18:29:08.714328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.714117 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" Oct 30 18:29:08.714328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.714141 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" event={"ID":"ba535e0e-ea07-4d37-8c4c-747b060a62dd","Type":"ContainerDied","Data":"1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9"} Oct 30 18:29:08.714328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.714214 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls" event={"ID":"ba535e0e-ea07-4d37-8c4c-747b060a62dd","Type":"ContainerDied","Data":"dda31befb0c5cd5dbd79f8b3eec69aaacb3369ffd1ea94b773eba2e69d769b48"} Oct 30 18:29:08.714328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.714252 4862 scope.go:122] "RemoveContainer" containerID="1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9" Oct 30 18:29:08.731634 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.730645 4862 scope.go:122] "RemoveContainer" containerID="21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379" Oct 30 18:29:08.744510 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.744480 4862 scope.go:122] "RemoveContainer" containerID="1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9" Oct 30 18:29:08.744832 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:29:08.744792 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9\": container with ID starting with 1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9 not found: ID does not exist" containerID="1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9" Oct 30 18:29:08.744901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.744848 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9"} err="failed to get container status \"1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9\": rpc error: code = NotFound desc = could not find container \"1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9\": container with ID starting with 1772b6ae2400ce2d0dbee076587decb2a499f1a4092189ae5588defb916898c9 not found: ID does not exist" Oct 30 18:29:08.744901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.744873 4862 scope.go:122] "RemoveContainer" containerID="21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379" Oct 30 18:29:08.745168 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:29:08.745147 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379\": container with ID starting with 21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379 not found: ID does not exist" containerID="21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379" Oct 30 18:29:08.745239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.745177 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379"} err="failed to get container status \"21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379\": rpc error: code = NotFound desc = could not find container \"21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379\": container with ID starting with 21a3c0a983e26b26bf5164bd9a13dc608270c9c02cc1bff538d418d87c417379 not found: ID does not exist" Oct 30 18:29:08.780146 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.780125 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls"] Oct 30 18:29:08.793848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:08.793822 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f826dd-openshift-default-6fc76ff86f-fcvls"] Oct 30 18:29:09.063803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:09.063770 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" path="/var/lib/kubelet/pods/ba535e0e-ea07-4d37-8c4c-747b060a62dd/volumes" Oct 30 18:29:18.832237 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:18.832204 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:29:19.879511 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:29:19.879468 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:29:21.643829 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:21.643795 4862 scope.go:122] "RemoveContainer" containerID="b3fbfc3714bfc125ad50125030b8476b74aef303e9345a193782afb69b8423b7" Oct 30 18:29:23.264092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264056 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb"] Oct 30 18:29:23.264542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264527 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" Oct 30 18:29:23.264585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264543 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" Oct 30 18:29:23.264585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264552 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b86e3b0b-0fe1-417e-b000-e3f84310f2f8" containerName="maas-api" Oct 30 18:29:23.264585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264557 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86e3b0b-0fe1-417e-b000-e3f84310f2f8" containerName="maas-api" Oct 30 18:29:23.264585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264569 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" Oct 30 18:29:23.264585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264574 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" Oct 30 18:29:23.264585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264583 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264587 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264594 4862 cpu_manager.go:414] "containerMap: removing container" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264599 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264605 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264609 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264617 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264621 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264629 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264634 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264642 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264646 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264653 4862 cpu_manager.go:414] "containerMap: removing container" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264658 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264668 4862 cpu_manager.go:414] "containerMap: removing container" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264672 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264678 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264683 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264692 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264696 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264709 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264713 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264722 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" Oct 30 18:29:23.264768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264726 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264817 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264824 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c89f09-f8de-4a56-9a63-e30e1e6f27aa" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264833 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264839 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="47113181-eed4-4806-afa0-a05250b7b7a8" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264844 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fc7558-2c7b-45af-ad4f-4091b840ee5c" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264849 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264853 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2613d9-58e1-438d-b256-d6bc303cd461" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264859 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264864 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4d1c63-04da-46e0-953e-40e5ef9663e1" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264872 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8b5882-396e-446e-9efc-b7c11e0d454a" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264877 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca9be35-a071-480e-97bb-68eedec2f8c6" containerName="istio-proxy" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264881 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86e3b0b-0fe1-417e-b000-e3f84310f2f8" containerName="maas-api" Oct 30 18:29:23.265442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.264886 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ae0d79-41b8-4c59-bcb4-225820e3b21d" containerName="istio-proxy" Oct 30 18:29:23.269469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.269452 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.293534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.293505 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb"] Oct 30 18:29:23.414138 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.414103 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.414138 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.414143 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.414346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.414170 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.414346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.414305 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hzm\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-kube-api-access-49hzm\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.414346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.414340 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.414502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.414451 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.414552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.414506 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca034077-40ef-403c-ad53-f2a0b34524cc-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.414614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.414567 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.414653 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.414620 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.515614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.515544 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.515614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.515574 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.515614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.515596 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.515867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.515707 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hzm\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-kube-api-access-49hzm\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.515867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.515768 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.515867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.515842 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.516021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.515877 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca034077-40ef-403c-ad53-f2a0b34524cc-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.516021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.515984 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-certs\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.516021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.516012 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.516174 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.516058 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.516333 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.516310 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-credential-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.516477 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.516393 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-data\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.516541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.516449 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-socket\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.516687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.516661 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca034077-40ef-403c-ad53-f2a0b34524cc-istiod-ca-cert\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.518151 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.518127 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-envoy\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.518801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.518781 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-podinfo\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.536364 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.536342 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-token\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.536490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.536476 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hzm\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-kube-api-access-49hzm\") pod \"e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.583726 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.583702 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:23.899627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.899600 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb"] Oct 30 18:29:23.901077 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:29:23.901047 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca034077_40ef_403c_ad53_f2a0b34524cc.slice/crio-10361fcb4387e216706f4f02d0e633e81dd4d00d7f7c9bd78e5d6577e6508dcc WatchSource:0}: Error finding container 10361fcb4387e216706f4f02d0e633e81dd4d00d7f7c9bd78e5d6577e6508dcc: Status 404 returned error can't find the container with id 10361fcb4387e216706f4f02d0e633e81dd4d00d7f7c9bd78e5d6577e6508dcc Oct 30 18:29:23.903465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.903437 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:23.903537 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.903507 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:23.903574 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:23.903538 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:24.820917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:24.820884 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" event={"ID":"ca034077-40ef-403c-ad53-f2a0b34524cc","Type":"ContainerStarted","Data":"ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68"} Oct 30 18:29:24.821467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:24.820943 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" event={"ID":"ca034077-40ef-403c-ad53-f2a0b34524cc","Type":"ContainerStarted","Data":"10361fcb4387e216706f4f02d0e633e81dd4d00d7f7c9bd78e5d6577e6508dcc"} Oct 30 18:29:24.875395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:24.875347 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" podStartSLOduration=1.875332427 podStartE2EDuration="1.875332427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:24.874075463 +0000 UTC m=+1863.923120376" watchObservedRunningTime="2026-10-30 18:29:24.875332427 +0000 UTC m=+1863.924377398" Oct 30 18:29:25.584336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:25.584303 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:25.586029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:25.585999 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:29:25.586167 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:25.586045 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 18:29:26.584481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:26.584449 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:29:26.584858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:26.584499 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 18:29:27.588679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:27.588642 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:29:27.589292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:27.589226 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 18:29:28.584492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:28.584458 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:29:28.584679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:28.584507 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 18:29:29.584636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:29.584605 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:29:29.585097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:29.584664 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 18:29:30.585006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:30.584977 4862 patch_prober.go:29] interesting pod/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:29:30.585392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:30.585029 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 18:29:31.588598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:31.588569 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:31.589080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:31.588938 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:31.589551 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:31.589532 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:29:38.230683 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.230648 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml"] Oct 30 18:29:38.231183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.231170 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1735e3-e2ea-4e64-a372-e88738de6b4a" containerName="istio-proxy" Oct 30 18:29:38.231227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.231194 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba535e0e-ea07-4d37-8c4c-747b060a62dd" containerName="istio-proxy" Oct 30 18:29:38.234837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.234820 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.277776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.277730 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml"] Oct 30 18:29:38.308073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.308035 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.308289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.308122 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8hjn\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-kube-api-access-j8hjn\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.308289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.308162 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.308289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.308201 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de62f70d-94b4-4121-9095-efbda0423381-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.308485 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.308290 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.308485 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.308346 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.308675 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.308650 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.308791 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.308697 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.308871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.308786 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de62f70d-94b4-4121-9095-efbda0423381-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410130 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410214 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410247 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410318 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de62f70d-94b4-4121-9095-efbda0423381-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410397 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410495 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8hjn\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-kube-api-access-j8hjn\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410589 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410634 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410884 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410688 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410884 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410726 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de62f70d-94b4-4121-9095-efbda0423381-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410884 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410780 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.410884 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410865 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.411091 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.410910 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.411436 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.411418 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de62f70d-94b4-4121-9095-efbda0423381-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.413222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.413193 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.413342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.413219 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de62f70d-94b4-4121-9095-efbda0423381-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.430769 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.430742 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.431110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.431093 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8hjn\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-kube-api-access-j8hjn\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.550087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.550066 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:38.867954 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.867925 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml"] Oct 30 18:29:38.869049 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:29:38.869029 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde62f70d_94b4_4121_9095_efbda0423381.slice/crio-9eb01ea42112e92c9109e9f9d99116001f89d131054c1d83bdec8d9c8a75d58d WatchSource:0}: Error finding container 9eb01ea42112e92c9109e9f9d99116001f89d131054c1d83bdec8d9c8a75d58d: Status 404 returned error can't find the container with id 9eb01ea42112e92c9109e9f9d99116001f89d131054c1d83bdec8d9c8a75d58d Oct 30 18:29:38.871499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.871468 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:38.871568 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.871535 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:38.871612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.871565 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:38.925403 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:38.925374 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" event={"ID":"de62f70d-94b4-4121-9095-efbda0423381","Type":"ContainerStarted","Data":"9eb01ea42112e92c9109e9f9d99116001f89d131054c1d83bdec8d9c8a75d58d"} Oct 30 18:29:39.964289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:39.958870 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" event={"ID":"de62f70d-94b4-4121-9095-efbda0423381","Type":"ContainerStarted","Data":"4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2"} Oct 30 18:29:40.048287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:40.047061 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" podStartSLOduration=2.047039704 podStartE2EDuration="2.047039704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:40.024319681 +0000 UTC m=+1879.073364594" watchObservedRunningTime="2026-10-30 18:29:40.047039704 +0000 UTC m=+1879.096084620" Oct 30 18:29:40.567289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:40.566252 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:40.590286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:40.568795 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:29:40.590286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:40.568868 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:29:41.552614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:41.552581 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:29:41.553984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:41.553162 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:29:42.552288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:42.551617 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:29:42.552288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:42.551676 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:29:43.550808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:43.550773 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:29:43.551436 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:43.551403 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:29:44.552664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:44.552616 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:29:44.553179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:44.552691 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:29:45.550814 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:45.550779 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:29:45.551014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:45.550834 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:29:47.492384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:47.492338 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:47.507294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:47.497201 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:47.507294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:47.502485 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:29:49.868827 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:29:49.868793 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:29:53.236856 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.236818 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf"] Oct 30 18:29:53.241155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.241131 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.270901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.270865 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d91bf0-629e-4d73-a340-36d38891b3b2-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.271066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.270922 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.271125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.271057 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.271125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.271090 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.271125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.271118 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.271322 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.271228 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8v5\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-kube-api-access-cp8v5\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.271442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.271421 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.271503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.271487 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.271559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.271518 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.273767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.273740 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf"] Oct 30 18:29:53.372148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372115 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d91bf0-629e-4d73-a340-36d38891b3b2-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372157 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372218 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372241 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372294 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372327 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8v5\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-kube-api-access-cp8v5\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372580 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372464 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372580 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372528 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372580 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372550 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372737 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-certs\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372828 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-data\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372920 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372906 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-credential-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.372983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372965 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d91bf0-629e-4d73-a340-36d38891b3b2-istiod-ca-cert\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.373039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.372999 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-socket\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.374899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.374875 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-envoy\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.375006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.374988 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-podinfo\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.391697 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.391666 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-token\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.391827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.391749 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8v5\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-kube-api-access-cp8v5\") pod \"e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.557432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.557395 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:53.882353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.882331 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf"] Oct 30 18:29:53.884619 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:29:53.884593 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d91bf0_629e_4d73_a340_36d38891b3b2.slice/crio-f11d8235f24bde87d1bf94a584943024ea5370f2e844cbd3effe7ab8b2a88f6d WatchSource:0}: Error finding container f11d8235f24bde87d1bf94a584943024ea5370f2e844cbd3effe7ab8b2a88f6d: Status 404 returned error can't find the container with id f11d8235f24bde87d1bf94a584943024ea5370f2e844cbd3effe7ab8b2a88f6d Oct 30 18:29:53.887549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.887518 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:53.887633 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.887584 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:53.887633 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:53.887612 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:29:54.056376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:54.056328 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" event={"ID":"11d91bf0-629e-4d73-a340-36d38891b3b2","Type":"ContainerStarted","Data":"d18dff880c6ea486ac959cdafb6c73f2ffb7e0ec47b13faa7685b5522f12e531"} Oct 30 18:29:54.056535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:54.056395 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" event={"ID":"11d91bf0-629e-4d73-a340-36d38891b3b2","Type":"ContainerStarted","Data":"f11d8235f24bde87d1bf94a584943024ea5370f2e844cbd3effe7ab8b2a88f6d"} Oct 30 18:29:54.098335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:54.098285 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podStartSLOduration=1.098257195 podStartE2EDuration="1.098257195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:54.096933814 +0000 UTC m=+1893.145978731" watchObservedRunningTime="2026-10-30 18:29:54.098257195 +0000 UTC m=+1893.147302108" Oct 30 18:29:54.558252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:54.558220 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:29:54.560053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:54.560027 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:29:54.560202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:54.560090 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 18:29:55.558287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:55.557794 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:29:55.558287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:55.557853 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 18:29:56.559573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:56.559538 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:29:56.560007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:56.559623 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 18:29:57.558206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:57.558174 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:29:57.558478 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:57.558445 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 18:29:58.558325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:58.558289 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:29:58.558807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:58.558363 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 18:29:59.209490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:59.209444 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:29:59.561417 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:59.559573 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:29:59.561417 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:29:59.559630 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 18:30:00.163065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.163029 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95"] Oct 30 18:30:00.178768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.178743 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:00.179014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.178940 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95"] Oct 30 18:30:00.184982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.184951 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792"] Oct 30 18:30:00.200282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.193653 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:30:00.212065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.210383 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792"] Oct 30 18:30:00.212065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.210466 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:00.287486 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.287460 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf46da3-ec26-4009-accc-6f51e5572648-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-td792\" (UID: \"eaf46da3-ec26-4009-accc-6f51e5572648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:00.287698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.287683 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpspt\" (UniqueName: \"kubernetes.io/projected/0accdf56-d48e-4d52-9f10-004582465537-kube-api-access-jpspt\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95\" (UID: \"0accdf56-d48e-4d52-9f10-004582465537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:00.287839 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.287828 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0accdf56-d48e-4d52-9f10-004582465537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95\" (UID: \"0accdf56-d48e-4d52-9f10-004582465537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:00.288006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.287985 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvmx\" (UniqueName: \"kubernetes.io/projected/eaf46da3-ec26-4009-accc-6f51e5572648-kube-api-access-whvmx\") pod \"maas-api-key-cleanup-29889750-td792\" (UID: \"eaf46da3-ec26-4009-accc-6f51e5572648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:00.393286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.391740 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvmx\" (UniqueName: \"kubernetes.io/projected/eaf46da3-ec26-4009-accc-6f51e5572648-kube-api-access-whvmx\") pod \"maas-api-key-cleanup-29889750-td792\" (UID: \"eaf46da3-ec26-4009-accc-6f51e5572648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:00.393286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.391835 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf46da3-ec26-4009-accc-6f51e5572648-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-td792\" (UID: \"eaf46da3-ec26-4009-accc-6f51e5572648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:00.393286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.391878 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpspt\" (UniqueName: \"kubernetes.io/projected/0accdf56-d48e-4d52-9f10-004582465537-kube-api-access-jpspt\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95\" (UID: \"0accdf56-d48e-4d52-9f10-004582465537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:00.393286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.391983 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0accdf56-d48e-4d52-9f10-004582465537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95\" (UID: \"0accdf56-d48e-4d52-9f10-004582465537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:00.393286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.392829 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0accdf56-d48e-4d52-9f10-004582465537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95\" (UID: \"0accdf56-d48e-4d52-9f10-004582465537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:00.395384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.393704 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf46da3-ec26-4009-accc-6f51e5572648-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-td792\" (UID: \"eaf46da3-ec26-4009-accc-6f51e5572648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:00.427036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.426962 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvmx\" (UniqueName: \"kubernetes.io/projected/eaf46da3-ec26-4009-accc-6f51e5572648-kube-api-access-whvmx\") pod \"maas-api-key-cleanup-29889750-td792\" (UID: \"eaf46da3-ec26-4009-accc-6f51e5572648\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:00.429026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.429000 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpspt\" (UniqueName: \"kubernetes.io/projected/0accdf56-d48e-4d52-9f10-004582465537-kube-api-access-jpspt\") pod \"maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95\" (UID: \"0accdf56-d48e-4d52-9f10-004582465537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:00.498465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.498428 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:00.544749 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.544478 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:00.562058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.560755 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:30:00.562058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:00.560809 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 18:30:01.335028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:01.334997 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95"] Oct 30 18:30:01.646288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:01.632349 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:01.646288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:01.633471 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 18:30:01.646288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:01.634945 4862 patch_prober.go:29] interesting pod/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:30:01.646288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:01.634985 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 18:30:01.747287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:01.743137 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.004442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:02.003489 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792"] Oct 30 18:30:02.189640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:02.189245 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:02.191470 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:02.190659 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 18:30:02.362474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:02.359491 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" event={"ID":"eaf46da3-ec26-4009-accc-6f51e5572648","Type":"ContainerStarted","Data":"998d58109efe33325c8419514f142a5f006ac5236fdf6fcf69b36db8e9709ca0"} Oct 30 18:30:02.374527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:02.369193 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" event={"ID":"0accdf56-d48e-4d52-9f10-004582465537","Type":"ContainerStarted","Data":"a30ca976f386f38cba9269a9f559cbe2fdd57eda8bbbc0b7ee0ba1f66f891301"} Oct 30 18:30:02.944749 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:02.944655 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:30:02.945224 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:02.944964 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:30:02.945858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:02.945836 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:30:03.376982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:03.376954 4862 generic.go:356] "Generic (PLEG): container finished" podID="0accdf56-d48e-4d52-9f10-004582465537" containerID="7d85498e4d08421dd2a7d2f833d6aab1a9337df7949b6e487dbce097aa0940cc" exitCode=60 Oct 30 18:30:03.377111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:03.377034 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" event={"ID":"0accdf56-d48e-4d52-9f10-004582465537","Type":"ContainerDied","Data":"7d85498e4d08421dd2a7d2f833d6aab1a9337df7949b6e487dbce097aa0940cc"} Oct 30 18:30:03.377329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:03.377306 4862 scope.go:122] "RemoveContainer" containerID="7d85498e4d08421dd2a7d2f833d6aab1a9337df7949b6e487dbce097aa0940cc" Oct 30 18:30:03.379021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:03.378997 4862 generic.go:356] "Generic (PLEG): container finished" podID="eaf46da3-ec26-4009-accc-6f51e5572648" containerID="b9b830e01b883130dd2de78ce614f7eb36f47a9c8fb6a0a1fc521d8617411308" exitCode=60 Oct 30 18:30:03.379104 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:03.379044 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" event={"ID":"eaf46da3-ec26-4009-accc-6f51e5572648","Type":"ContainerDied","Data":"b9b830e01b883130dd2de78ce614f7eb36f47a9c8fb6a0a1fc521d8617411308"} Oct 30 18:30:03.379335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:03.379316 4862 scope.go:122] "RemoveContainer" containerID="b9b830e01b883130dd2de78ce614f7eb36f47a9c8fb6a0a1fc521d8617411308" Oct 30 18:30:04.388880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:04.388846 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" event={"ID":"0accdf56-d48e-4d52-9f10-004582465537","Type":"ContainerStarted","Data":"49113e0791d12cea22affdc987711226a11197ccfe05e569a51e5f572f95596b"} Oct 30 18:30:04.391013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:04.390983 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" event={"ID":"eaf46da3-ec26-4009-accc-6f51e5572648","Type":"ContainerStarted","Data":"2a9ebeb08076391144c4adf66ff0843f65ab987d76fe111ed57403edb156cb07"} Oct 30 18:30:04.429449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:04.429142 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" podStartSLOduration=4.429122015 podStartE2EDuration="4.429122015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:04.427596915 +0000 UTC m=+1903.476641829" watchObservedRunningTime="2026-10-30 18:30:04.429122015 +0000 UTC m=+1903.478166930" Oct 30 18:30:04.467004 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:04.466964 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" podStartSLOduration=4.466950902 podStartE2EDuration="4.466950902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:04.465103281 +0000 UTC m=+1903.514148193" watchObservedRunningTime="2026-10-30 18:30:04.466950902 +0000 UTC m=+1903.515995849" Oct 30 18:30:05.399923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:05.399885 4862 generic.go:356] "Generic (PLEG): container finished" podID="eaf46da3-ec26-4009-accc-6f51e5572648" containerID="2a9ebeb08076391144c4adf66ff0843f65ab987d76fe111ed57403edb156cb07" exitCode=60 Oct 30 18:30:05.400435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:05.399952 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" event={"ID":"eaf46da3-ec26-4009-accc-6f51e5572648","Type":"ContainerDied","Data":"2a9ebeb08076391144c4adf66ff0843f65ab987d76fe111ed57403edb156cb07"} Oct 30 18:30:05.400435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:05.400002 4862 scope.go:122] "RemoveContainer" containerID="b9b830e01b883130dd2de78ce614f7eb36f47a9c8fb6a0a1fc521d8617411308" Oct 30 18:30:05.400435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:05.400225 4862 scope.go:122] "RemoveContainer" containerID="2a9ebeb08076391144c4adf66ff0843f65ab987d76fe111ed57403edb156cb07" Oct 30 18:30:05.400595 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:30:05.400489 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-td792_odh-ai-gateway-infra(eaf46da3-ec26-4009-accc-6f51e5572648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" Oct 30 18:30:05.401940 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:05.401913 4862 generic.go:356] "Generic (PLEG): container finished" podID="0accdf56-d48e-4d52-9f10-004582465537" containerID="49113e0791d12cea22affdc987711226a11197ccfe05e569a51e5f572f95596b" exitCode=60 Oct 30 18:30:05.402032 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:05.401991 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" event={"ID":"0accdf56-d48e-4d52-9f10-004582465537","Type":"ContainerDied","Data":"49113e0791d12cea22affdc987711226a11197ccfe05e569a51e5f572f95596b"} Oct 30 18:30:05.402220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:05.402204 4862 scope.go:122] "RemoveContainer" containerID="49113e0791d12cea22affdc987711226a11197ccfe05e569a51e5f572f95596b" Oct 30 18:30:05.402455 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:30:05.402438 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95_odh-ai-gateway-infra(0accdf56-d48e-4d52-9f10-004582465537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" podUID="0accdf56-d48e-4d52-9f10-004582465537" Oct 30 18:30:05.412396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:05.412380 4862 scope.go:122] "RemoveContainer" containerID="7d85498e4d08421dd2a7d2f833d6aab1a9337df7949b6e487dbce097aa0940cc" Oct 30 18:30:06.410486 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:06.410456 4862 scope.go:122] "RemoveContainer" containerID="49113e0791d12cea22affdc987711226a11197ccfe05e569a51e5f572f95596b" Oct 30 18:30:06.410903 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:30:06.410735 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95_odh-ai-gateway-infra(0accdf56-d48e-4d52-9f10-004582465537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" podUID="0accdf56-d48e-4d52-9f10-004582465537" Oct 30 18:30:06.412086 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:06.412065 4862 scope.go:122] "RemoveContainer" containerID="2a9ebeb08076391144c4adf66ff0843f65ab987d76fe111ed57403edb156cb07" Oct 30 18:30:06.412329 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:30:06.412310 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-td792_odh-ai-gateway-infra(eaf46da3-ec26-4009-accc-6f51e5572648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" Oct 30 18:30:16.055001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:16.054971 4862 scope.go:122] "RemoveContainer" containerID="2a9ebeb08076391144c4adf66ff0843f65ab987d76fe111ed57403edb156cb07" Oct 30 18:30:16.492573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:16.492534 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" event={"ID":"eaf46da3-ec26-4009-accc-6f51e5572648","Type":"ContainerStarted","Data":"ac175b2db97ab67ca49c64ea7c7cec9bc310fbed20acb38c1576b17a1153ffd3"} Oct 30 18:30:17.500896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:17.500861 4862 generic.go:356] "Generic (PLEG): container finished" podID="eaf46da3-ec26-4009-accc-6f51e5572648" containerID="ac175b2db97ab67ca49c64ea7c7cec9bc310fbed20acb38c1576b17a1153ffd3" exitCode=60 Oct 30 18:30:17.501315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:17.500935 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" event={"ID":"eaf46da3-ec26-4009-accc-6f51e5572648","Type":"ContainerDied","Data":"ac175b2db97ab67ca49c64ea7c7cec9bc310fbed20acb38c1576b17a1153ffd3"} Oct 30 18:30:17.501315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:17.500997 4862 scope.go:122] "RemoveContainer" containerID="2a9ebeb08076391144c4adf66ff0843f65ab987d76fe111ed57403edb156cb07" Oct 30 18:30:17.501440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:17.501381 4862 scope.go:122] "RemoveContainer" containerID="ac175b2db97ab67ca49c64ea7c7cec9bc310fbed20acb38c1576b17a1153ffd3" Oct 30 18:30:17.501711 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:30:17.501680 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-td792_odh-ai-gateway-infra(eaf46da3-ec26-4009-accc-6f51e5572648)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" Oct 30 18:30:17.540876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:17.540854 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792"] Oct 30 18:30:18.055323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:18.055298 4862 scope.go:122] "RemoveContainer" containerID="49113e0791d12cea22affdc987711226a11197ccfe05e569a51e5f572f95596b" Oct 30 18:30:18.511180 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:18.511153 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" event={"ID":"0accdf56-d48e-4d52-9f10-004582465537","Type":"ContainerStarted","Data":"d3f77567cbd1cb04a6c985c447b1fbdc32e4940cd7697aac577756c2d24a0f0b"} Oct 30 18:30:18.682575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:18.682549 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:18.797964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:18.797933 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf46da3-ec26-4009-accc-6f51e5572648-kube-api-access-whvmx\" (UniqueName: \"kubernetes.io/projected/eaf46da3-ec26-4009-accc-6f51e5572648-kube-api-access-whvmx\") pod \"eaf46da3-ec26-4009-accc-6f51e5572648\" (UID: \"eaf46da3-ec26-4009-accc-6f51e5572648\") " Oct 30 18:30:18.798125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:18.798035 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf46da3-ec26-4009-accc-6f51e5572648-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf46da3-ec26-4009-accc-6f51e5572648-maas-api-service-ca\") pod \"eaf46da3-ec26-4009-accc-6f51e5572648\" (UID: \"eaf46da3-ec26-4009-accc-6f51e5572648\") " Oct 30 18:30:18.798406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:18.798382 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf46da3-ec26-4009-accc-6f51e5572648-maas-api-service-ca" pod "eaf46da3-ec26-4009-accc-6f51e5572648" (UID: "eaf46da3-ec26-4009-accc-6f51e5572648"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:18.800216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:18.800186 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf46da3-ec26-4009-accc-6f51e5572648-kube-api-access-whvmx" pod "eaf46da3-ec26-4009-accc-6f51e5572648" (UID: "eaf46da3-ec26-4009-accc-6f51e5572648"). InnerVolumeSpecName "kube-api-access-whvmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:18.899067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:18.899041 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf46da3-ec26-4009-accc-6f51e5572648-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:30:18.899067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:18.899068 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvmx\" (UniqueName: \"kubernetes.io/projected/eaf46da3-ec26-4009-accc-6f51e5572648-kube-api-access-whvmx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:30:19.524298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.524240 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" event={"ID":"eaf46da3-ec26-4009-accc-6f51e5572648","Type":"ContainerDied","Data":"998d58109efe33325c8419514f142a5f006ac5236fdf6fcf69b36db8e9709ca0"} Oct 30 18:30:19.524954 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.524920 4862 scope.go:122] "RemoveContainer" containerID="ac175b2db97ab67ca49c64ea7c7cec9bc310fbed20acb38c1576b17a1153ffd3" Oct 30 18:30:19.525079 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.524321 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792" Oct 30 18:30:19.528760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.528727 4862 generic.go:356] "Generic (PLEG): container finished" podID="0accdf56-d48e-4d52-9f10-004582465537" containerID="d3f77567cbd1cb04a6c985c447b1fbdc32e4940cd7697aac577756c2d24a0f0b" exitCode=60 Oct 30 18:30:19.528881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.528796 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" event={"ID":"0accdf56-d48e-4d52-9f10-004582465537","Type":"ContainerDied","Data":"d3f77567cbd1cb04a6c985c447b1fbdc32e4940cd7697aac577756c2d24a0f0b"} Oct 30 18:30:19.529108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.529095 4862 scope.go:122] "RemoveContainer" containerID="d3f77567cbd1cb04a6c985c447b1fbdc32e4940cd7697aac577756c2d24a0f0b" Oct 30 18:30:19.529381 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:30:19.529358 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95_odh-ai-gateway-infra(0accdf56-d48e-4d52-9f10-004582465537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" podUID="0accdf56-d48e-4d52-9f10-004582465537" Oct 30 18:30:19.535683 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.535668 4862 scope.go:122] "RemoveContainer" containerID="49113e0791d12cea22affdc987711226a11197ccfe05e569a51e5f572f95596b" Oct 30 18:30:19.576741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.576719 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792"] Oct 30 18:30:19.579806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.579786 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-td792"] Oct 30 18:30:19.594688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:19.594668 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95"] Oct 30 18:30:19.870522 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:30:19.870489 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:30:20.697710 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:20.697683 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:20.819610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:20.819581 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0accdf56-d48e-4d52-9f10-004582465537-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0accdf56-d48e-4d52-9f10-004582465537-maas-api-service-ca\") pod \"0accdf56-d48e-4d52-9f10-004582465537\" (UID: \"0accdf56-d48e-4d52-9f10-004582465537\") " Oct 30 18:30:20.819776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:20.819631 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0accdf56-d48e-4d52-9f10-004582465537-kube-api-access-jpspt\" (UniqueName: \"kubernetes.io/projected/0accdf56-d48e-4d52-9f10-004582465537-kube-api-access-jpspt\") pod \"0accdf56-d48e-4d52-9f10-004582465537\" (UID: \"0accdf56-d48e-4d52-9f10-004582465537\") " Oct 30 18:30:20.819996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:20.819969 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0accdf56-d48e-4d52-9f10-004582465537-maas-api-service-ca" pod "0accdf56-d48e-4d52-9f10-004582465537" (UID: "0accdf56-d48e-4d52-9f10-004582465537"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:20.820144 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:20.820129 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0accdf56-d48e-4d52-9f10-004582465537-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:30:20.821843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:20.821820 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0accdf56-d48e-4d52-9f10-004582465537-kube-api-access-jpspt" pod "0accdf56-d48e-4d52-9f10-004582465537" (UID: "0accdf56-d48e-4d52-9f10-004582465537"). InnerVolumeSpecName "kube-api-access-jpspt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:20.920994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:20.920958 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpspt\" (UniqueName: \"kubernetes.io/projected/0accdf56-d48e-4d52-9f10-004582465537-kube-api-access-jpspt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:30:21.061963 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:21.061930 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" path="/var/lib/kubelet/pods/eaf46da3-ec26-4009-accc-6f51e5572648/volumes" Oct 30 18:30:21.544706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:21.544676 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" Oct 30 18:30:21.544877 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:21.544726 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95" event={"ID":"0accdf56-d48e-4d52-9f10-004582465537","Type":"ContainerDied","Data":"a30ca976f386f38cba9269a9f559cbe2fdd57eda8bbbc0b7ee0ba1f66f891301"} Oct 30 18:30:21.544877 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:21.544786 4862 scope.go:122] "RemoveContainer" containerID="d3f77567cbd1cb04a6c985c447b1fbdc32e4940cd7697aac577756c2d24a0f0b" Oct 30 18:30:21.596080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:21.596053 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95"] Oct 30 18:30:21.599393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:21.599371 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-380c08-29889750-vnl95"] Oct 30 18:30:22.509981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:22.509952 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:30:23.065066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.064906 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0accdf56-d48e-4d52-9f10-004582465537" path="/var/lib/kubelet/pods/0accdf56-d48e-4d52-9f10-004582465537/volumes" Oct 30 18:30:23.268317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.268283 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq"] Oct 30 18:30:23.268916 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.268885 4862 cpu_manager.go:414] "containerMap: removing container" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" containerName="cleanup" Oct 30 18:30:23.269056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.268920 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" containerName="cleanup" Oct 30 18:30:23.269056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.268943 4862 cpu_manager.go:414] "containerMap: removing container" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" containerName="cleanup" Oct 30 18:30:23.269056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.268953 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" containerName="cleanup" Oct 30 18:30:23.269056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.268997 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0accdf56-d48e-4d52-9f10-004582465537" containerName="cleanup" Oct 30 18:30:23.269056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.269007 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0accdf56-d48e-4d52-9f10-004582465537" containerName="cleanup" Oct 30 18:30:23.269056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.269027 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0accdf56-d48e-4d52-9f10-004582465537" containerName="cleanup" Oct 30 18:30:23.269056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.269036 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0accdf56-d48e-4d52-9f10-004582465537" containerName="cleanup" Oct 30 18:30:23.269328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.269071 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0accdf56-d48e-4d52-9f10-004582465537" containerName="cleanup" Oct 30 18:30:23.269328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.269080 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0accdf56-d48e-4d52-9f10-004582465537" containerName="cleanup" Oct 30 18:30:23.269328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.269294 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" containerName="cleanup" Oct 30 18:30:23.269328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.269312 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" containerName="cleanup" Oct 30 18:30:23.269328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.269327 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0accdf56-d48e-4d52-9f10-004582465537" containerName="cleanup" Oct 30 18:30:23.269480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.269337 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0accdf56-d48e-4d52-9f10-004582465537" containerName="cleanup" Oct 30 18:30:23.274548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.274523 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.302026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.301991 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq"] Oct 30 18:30:23.359336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.359177 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.359336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.359209 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b0d537-406a-479c-8deb-6f2afdfdce5a-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.359336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.359304 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.359620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.359405 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.359620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.359471 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.359620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.359508 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.359620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.359526 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.359620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.359596 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7q2\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-kube-api-access-jf7q2\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.359620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.359619 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461136 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461093 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461146 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b0d537-406a-479c-8deb-6f2afdfdce5a-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461201 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461317 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461412 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461453 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461480 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461519 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7q2\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-kube-api-access-jf7q2\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461547 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461731 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.461998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.461973 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.462536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.462040 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b0d537-406a-479c-8deb-6f2afdfdce5a-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.462656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.462372 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.462656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.462399 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.464133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.464105 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.464327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.464308 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.485867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.485487 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7q2\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-kube-api-access-jf7q2\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.486043 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.485932 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:23.612337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:23.611525 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:24.629598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:24.629565 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq"] Oct 30 18:30:24.665843 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:30:24.665808 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b0d537_406a_479c_8deb_6f2afdfdce5a.slice/crio-4d58b0e4ec84427f54fde920fb2c2a5bc235f0ad1749990075b9e14938382a4e WatchSource:0}: Error finding container 4d58b0e4ec84427f54fde920fb2c2a5bc235f0ad1749990075b9e14938382a4e: Status 404 returned error can't find the container with id 4d58b0e4ec84427f54fde920fb2c2a5bc235f0ad1749990075b9e14938382a4e Oct 30 18:30:24.681342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:24.681047 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:24.681342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:24.681169 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:24.681342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:24.681218 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:30:25.181344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:25.180394 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:30:25.202292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:25.201779 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 18:30:25.630293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:25.628318 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" event={"ID":"30b0d537-406a-479c-8deb-6f2afdfdce5a","Type":"ContainerStarted","Data":"d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30"} Oct 30 18:30:25.630293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:25.628385 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" event={"ID":"30b0d537-406a-479c-8deb-6f2afdfdce5a","Type":"ContainerStarted","Data":"4d58b0e4ec84427f54fde920fb2c2a5bc235f0ad1749990075b9e14938382a4e"} Oct 30 18:30:25.730289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:25.727993 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" podStartSLOduration=2.7279676889999998 podStartE2EDuration="2.727967689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:25.701329981 +0000 UTC m=+1924.750374896" watchObservedRunningTime="2026-10-30 18:30:25.727967689 +0000 UTC m=+1924.777012605" Oct 30 18:30:26.612186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:26.612152 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:26.613983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:26.613953 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:30:26.614132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:26.614004 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:30:27.613113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:27.613081 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:30:27.613553 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:27.613136 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:30:28.612213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:28.612174 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:30:28.612396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:28.612233 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:30:29.612456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:29.612422 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:30:29.612937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:29.612490 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:30:30.612647 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:30.612609 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:30:30.613074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:30.612672 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:30:31.612596 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:31.612519 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:30:31.612596 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:31.612583 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 18:30:32.766475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:32.766435 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:32.766966 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:32.766834 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:32.767932 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:32.767912 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:30:49.870370 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:30:49.870330 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:30:59.465835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:30:59.465802 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:31:19.876725 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:31:19.876622 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:31:38.319849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.319798 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs"] Oct 30 18:31:38.320592 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.320571 4862 cpu_manager.go:414] "containerMap: removing container" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" containerName="cleanup" Oct 30 18:31:38.320688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.320595 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" containerName="cleanup" Oct 30 18:31:38.320744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.320710 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf46da3-ec26-4009-accc-6f51e5572648" containerName="cleanup" Oct 30 18:31:38.320744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.320724 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0accdf56-d48e-4d52-9f10-004582465537" containerName="cleanup" Oct 30 18:31:38.324406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.324383 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.356872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.356840 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs"] Oct 30 18:31:38.458564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.458525 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.458740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.458585 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.458740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.458690 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.458740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.458730 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11730e43-29fb-4f70-b591-afed0868b014-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.458843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.458790 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11730e43-29fb-4f70-b591-afed0868b014-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.458843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.458812 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp269\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-kube-api-access-xp269\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.458957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.458944 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.458992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.458976 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.459026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.459019 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.559903 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.559867 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560104 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.559913 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560104 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.559950 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560104 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.559982 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.560100 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11730e43-29fb-4f70-b591-afed0868b014-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.560138 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11730e43-29fb-4f70-b591-afed0868b014-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.560167 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp269\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-kube-api-access-xp269\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.560431 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-credential-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.560488 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.560550 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560819 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.560797 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-socket\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.560997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.560975 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-certs\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.561086 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.561014 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11730e43-29fb-4f70-b591-afed0868b014-istiod-ca-cert\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.561226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.561208 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-data\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.562658 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.562635 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-envoy\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.563611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.563590 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11730e43-29fb-4f70-b591-afed0868b014-istio-podinfo\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.578931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.578865 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-istio-token\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.579095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.579074 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp269\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-kube-api-access-xp269\") pod \"e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.638767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.638735 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:38.954568 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.954542 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs"] Oct 30 18:31:38.955900 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:31:38.955864 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11730e43_29fb_4f70_b591_afed0868b014.slice/crio-12d9fba9b73c075dc1227757f45199069eec4c2409126662494077cfa9f87bc0 WatchSource:0}: Error finding container 12d9fba9b73c075dc1227757f45199069eec4c2409126662494077cfa9f87bc0: Status 404 returned error can't find the container with id 12d9fba9b73c075dc1227757f45199069eec4c2409126662494077cfa9f87bc0 Oct 30 18:31:38.958488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.958455 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:38.958540 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.958524 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:38.958575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:38.958552 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:31:39.223968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:39.223889 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" event={"ID":"11730e43-29fb-4f70-b591-afed0868b014","Type":"ContainerStarted","Data":"4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06"} Oct 30 18:31:39.223968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:39.223951 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" event={"ID":"11730e43-29fb-4f70-b591-afed0868b014","Type":"ContainerStarted","Data":"12d9fba9b73c075dc1227757f45199069eec4c2409126662494077cfa9f87bc0"} Oct 30 18:31:39.265609 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:39.265540 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" podStartSLOduration=1.265522027 podStartE2EDuration="1.265522027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:39.265152004 +0000 UTC m=+1998.314196928" watchObservedRunningTime="2026-10-30 18:31:39.265522027 +0000 UTC m=+1998.314566943" Oct 30 18:31:39.651675 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:39.644742 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:40.650287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:40.647150 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:31:40.650287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:40.647216 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:31:41.653620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:41.653587 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:31:41.654055 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:41.653641 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:31:42.653285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:42.651755 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:31:42.653285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:42.651828 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:31:43.650289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:43.648797 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:31:43.650289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:43.648853 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:31:44.642286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:44.641586 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:31:44.642286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:44.641644 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:31:45.639550 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:45.639518 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:31:45.639972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:45.639576 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:31:46.283382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:46.283345 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:31:46.706524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:46.706486 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:46.707014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:46.706759 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:46.708121 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:31:46.708096 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:31:49.872834 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:31:49.872794 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:32:09.000533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.000448 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw"] Oct 30 18:32:09.004883 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.004867 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:09.024455 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.024427 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw"] Oct 30 18:32:09.033308 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.033281 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5f7\" (UniqueName: \"kubernetes.io/projected/8d32fa3e-7c31-403d-bc9d-099927dae6e8-kube-api-access-ch5f7\") pod \"payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw\" (UID: \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:09.033434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.033395 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32fa3e-7c31-403d-bc9d-099927dae6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw\" (UID: \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:09.034052 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.034034 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-09ef1d\"" Oct 30 18:32:09.034108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.034037 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-09ef1d-dockercfg-k56dg\"" Oct 30 18:32:09.123287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.122052 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf"] Oct 30 18:32:09.129249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.129209 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf"] Oct 30 18:32:09.129778 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.129707 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:09.134894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.134867 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5f7\" (UniqueName: \"kubernetes.io/projected/8d32fa3e-7c31-403d-bc9d-099927dae6e8-kube-api-access-ch5f7\") pod \"payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw\" (UID: \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:09.136116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.136096 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32fa3e-7c31-403d-bc9d-099927dae6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw\" (UID: \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:09.137798 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.137778 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32fa3e-7c31-403d-bc9d-099927dae6e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw\" (UID: \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:09.205028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.204994 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5f7\" (UniqueName: \"kubernetes.io/projected/8d32fa3e-7c31-403d-bc9d-099927dae6e8-kube-api-access-ch5f7\") pod \"payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw\" (UID: \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:09.237232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.237189 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47b8\" (UniqueName: \"kubernetes.io/projected/d3bb768e-e677-44c7-a203-b02125cb3384-kube-api-access-l47b8\") pod \"payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf\" (UID: \"d3bb768e-e677-44c7-a203-b02125cb3384\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:09.237424 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.237386 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bb768e-e677-44c7-a203-b02125cb3384-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf\" (UID: \"d3bb768e-e677-44c7-a203-b02125cb3384\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:09.319389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.319361 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:09.339358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.339327 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47b8\" (UniqueName: \"kubernetes.io/projected/d3bb768e-e677-44c7-a203-b02125cb3384-kube-api-access-l47b8\") pod \"payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf\" (UID: \"d3bb768e-e677-44c7-a203-b02125cb3384\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:09.339496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.339461 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bb768e-e677-44c7-a203-b02125cb3384-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf\" (UID: \"d3bb768e-e677-44c7-a203-b02125cb3384\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:09.340388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.340357 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bb768e-e677-44c7-a203-b02125cb3384-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf\" (UID: \"d3bb768e-e677-44c7-a203-b02125cb3384\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:09.359086 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.359057 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47b8\" (UniqueName: \"kubernetes.io/projected/d3bb768e-e677-44c7-a203-b02125cb3384-kube-api-access-l47b8\") pod \"payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf\" (UID: \"d3bb768e-e677-44c7-a203-b02125cb3384\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:09.522152 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.522125 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:09.632544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.632511 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw"] Oct 30 18:32:09.638856 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:32:09.638809 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d32fa3e_7c31_403d_bc9d_099927dae6e8.slice/crio-6df8cee9ade39ae71557994aebb9409a4fc248535ceb4b6870dde8dc9ee4f648 WatchSource:0}: Error finding container 6df8cee9ade39ae71557994aebb9409a4fc248535ceb4b6870dde8dc9ee4f648: Status 404 returned error can't find the container with id 6df8cee9ade39ae71557994aebb9409a4fc248535ceb4b6870dde8dc9ee4f648 Oct 30 18:32:09.826869 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:09.826848 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf"] Oct 30 18:32:09.828756 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:32:09.828726 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bb768e_e677_44c7_a203_b02125cb3384.slice/crio-8efeec6498a59e5c342065e43b9a5b4f8326a96df50e917b787ad3838a47995d WatchSource:0}: Error finding container 8efeec6498a59e5c342065e43b9a5b4f8326a96df50e917b787ad3838a47995d: Status 404 returned error can't find the container with id 8efeec6498a59e5c342065e43b9a5b4f8326a96df50e917b787ad3838a47995d Oct 30 18:32:10.470562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.470525 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" event={"ID":"d3bb768e-e677-44c7-a203-b02125cb3384","Type":"ContainerStarted","Data":"97e686d078905234a44d7d5d989d9e825b2520e102d2513f6f8a5f83e4d82310"} Oct 30 18:32:10.471071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.470591 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" event={"ID":"d3bb768e-e677-44c7-a203-b02125cb3384","Type":"ContainerStarted","Data":"8efeec6498a59e5c342065e43b9a5b4f8326a96df50e917b787ad3838a47995d"} Oct 30 18:32:10.471071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.470730 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:10.472170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.472138 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" event={"ID":"8d32fa3e-7c31-403d-bc9d-099927dae6e8","Type":"ContainerStarted","Data":"c4e0f20eb14d79eab4151d1decd58777dc8db503ff3ddcf89223fbbf5c5610e6"} Oct 30 18:32:10.472342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.472186 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:10.472342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.472208 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" event={"ID":"8d32fa3e-7c31-403d-bc9d-099927dae6e8","Type":"ContainerStarted","Data":"6df8cee9ade39ae71557994aebb9409a4fc248535ceb4b6870dde8dc9ee4f648"} Oct 30 18:32:10.512712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.512628 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" podStartSLOduration=1.512615042 podStartE2EDuration="1.512615042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:10.51060903 +0000 UTC m=+2029.559653943" watchObservedRunningTime="2026-10-30 18:32:10.512615042 +0000 UTC m=+2029.561659955" Oct 30 18:32:10.552414 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.552356 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" podStartSLOduration=2.552340831 podStartE2EDuration="2.552340831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:10.548962179 +0000 UTC m=+2029.598007092" watchObservedRunningTime="2026-10-30 18:32:10.552340831 +0000 UTC m=+2029.601385744" Oct 30 18:32:10.761005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.760930 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88"] Oct 30 18:32:10.764999 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.764982 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:10.781190 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.781166 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-24b124\"" Oct 30 18:32:10.781949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.781927 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-24b124-dockercfg-52rlc\"" Oct 30 18:32:10.784585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.784563 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88"] Oct 30 18:32:10.858539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.857036 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbtj9\" (UniqueName: \"kubernetes.io/projected/56f556fe-3216-4078-af8d-920b06f41a7b-kube-api-access-sbtj9\") pod \"payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88\" (UID: \"56f556fe-3216-4078-af8d-920b06f41a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:10.858539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.857170 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f556fe-3216-4078-af8d-920b06f41a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88\" (UID: \"56f556fe-3216-4078-af8d-920b06f41a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:10.960317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.959089 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbtj9\" (UniqueName: \"kubernetes.io/projected/56f556fe-3216-4078-af8d-920b06f41a7b-kube-api-access-sbtj9\") pod \"payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88\" (UID: \"56f556fe-3216-4078-af8d-920b06f41a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:10.960317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.959216 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f556fe-3216-4078-af8d-920b06f41a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88\" (UID: \"56f556fe-3216-4078-af8d-920b06f41a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:10.960317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.960256 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f556fe-3216-4078-af8d-920b06f41a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88\" (UID: \"56f556fe-3216-4078-af8d-920b06f41a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:10.980559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:10.980449 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbtj9\" (UniqueName: \"kubernetes.io/projected/56f556fe-3216-4078-af8d-920b06f41a7b-kube-api-access-sbtj9\") pod \"payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88\" (UID: \"56f556fe-3216-4078-af8d-920b06f41a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:11.079064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:11.079023 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:11.381947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:11.381921 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88"] Oct 30 18:32:11.383614 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:32:11.383581 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f556fe_3216_4078_af8d_920b06f41a7b.slice/crio-d5fa70c8e429bb88954f15df45a27af79953115d37a09e1f0c6f0e4f66debcb1 WatchSource:0}: Error finding container d5fa70c8e429bb88954f15df45a27af79953115d37a09e1f0c6f0e4f66debcb1: Status 404 returned error can't find the container with id d5fa70c8e429bb88954f15df45a27af79953115d37a09e1f0c6f0e4f66debcb1 Oct 30 18:32:11.478907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:11.478866 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" event={"ID":"56f556fe-3216-4078-af8d-920b06f41a7b","Type":"ContainerStarted","Data":"cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030"} Oct 30 18:32:11.479255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:11.478928 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" event={"ID":"56f556fe-3216-4078-af8d-920b06f41a7b","Type":"ContainerStarted","Data":"d5fa70c8e429bb88954f15df45a27af79953115d37a09e1f0c6f0e4f66debcb1"} Oct 30 18:32:11.479319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:11.479251 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:11.519242 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:11.519200 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" podStartSLOduration=1.51918749 podStartE2EDuration="1.51918749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:11.51705023 +0000 UTC m=+2030.566095155" watchObservedRunningTime="2026-10-30 18:32:11.51918749 +0000 UTC m=+2030.568232432" Oct 30 18:32:19.872130 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:32:19.872089 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:32:21.481875 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:21.481843 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 18:32:21.482315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:21.481917 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 18:32:22.507056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:22.507020 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 18:32:27.324289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.322248 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv"] Oct 30 18:32:27.345310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.327344 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.354211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.354180 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv"] Oct 30 18:32:27.359373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.359348 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-d82e82-openshift-default-dockercfg-sc672\"" Oct 30 18:32:27.466560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.466524 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a7ed92-3470-4734-9896-3ce0def49be1-istiod-ca-cert\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.466746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.466578 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-envoy\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.466746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.466697 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xkw6\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-kube-api-access-5xkw6\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.466746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.466727 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.466909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.466825 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-token\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.466909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.466864 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-podinfo\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.466909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.466903 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-credential-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.467056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.466927 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-data\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.467056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.467004 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-certs\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.578134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.578036 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a7ed92-3470-4734-9896-3ce0def49be1-istiod-ca-cert\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.578498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.578473 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-envoy\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.578668 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.578652 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkw6\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-kube-api-access-5xkw6\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.578777 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.578763 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.578929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.578914 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-token\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.579054 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.579041 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-podinfo\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.579197 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.579184 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-credential-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.579328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.579314 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-data\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.579474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.579460 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-certs\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.580049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.580029 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-certs\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.580876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.580851 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a7ed92-3470-4734-9896-3ce0def49be1-istiod-ca-cert\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.587746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.587112 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.587746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.587469 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-credential-socket\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.587746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.587708 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-data\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.620032 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.609303 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-envoy\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.630669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.630631 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-token\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.632380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.631651 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-podinfo\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.640676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.640649 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkw6\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-kube-api-access-5xkw6\") pod \"e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:27.652321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:27.651503 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:28.011631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:28.011601 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv"] Oct 30 18:32:28.013881 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:32:28.013853 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a7ed92_3470_4734_9896_3ce0def49be1.slice/crio-1e722f90ffd167c6c24c01e595872e1be2db3d84e44097c6d1931a6653ea7c02 WatchSource:0}: Error finding container 1e722f90ffd167c6c24c01e595872e1be2db3d84e44097c6d1931a6653ea7c02: Status 404 returned error can't find the container with id 1e722f90ffd167c6c24c01e595872e1be2db3d84e44097c6d1931a6653ea7c02 Oct 30 18:32:28.017498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:28.017462 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:28.017652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:28.017543 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:28.017652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:28.017581 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:32:28.645387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:28.645355 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" event={"ID":"a4a7ed92-3470-4734-9896-3ce0def49be1","Type":"ContainerStarted","Data":"b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d"} Oct 30 18:32:28.645756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:28.645404 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" event={"ID":"a4a7ed92-3470-4734-9896-3ce0def49be1","Type":"ContainerStarted","Data":"1e722f90ffd167c6c24c01e595872e1be2db3d84e44097c6d1931a6653ea7c02"} Oct 30 18:32:28.651757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:28.651734 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:28.701502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:28.701448 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" podStartSLOduration=1.701432272 podStartE2EDuration="1.701432272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:28.699038274 +0000 UTC m=+2047.748083188" watchObservedRunningTime="2026-10-30 18:32:28.701432272 +0000 UTC m=+2047.750477183" Oct 30 18:32:29.657807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:29.657782 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:30.656552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:30.656516 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:30.657339 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:30.657318 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:32:43.543535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:32:43.543501 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:32:49.882410 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:32:49.882370 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:33:06.315494 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.315461 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg"] Oct 30 18:33:06.321228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.321205 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.356871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.356788 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1e0b31\"/\"kube-root-ca.crt\"" Oct 30 18:33:06.356871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.356834 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1e0b31\"/\"default-dockercfg-kpxm6\"" Oct 30 18:33:06.357080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.357014 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1e0b31\"/\"openshift-service-ca.crt\"" Oct 30 18:33:06.357138 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.357119 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1e0b31\"/\"autores-model-1e0b31-kserve-self-signed-certs\"" Oct 30 18:33:06.364303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.360247 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg"] Oct 30 18:33:06.456705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.456673 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l56cn\" (UniqueName: \"kubernetes.io/projected/4d81db87-2850-47b2-a927-7915d178933f-kube-api-access-l56cn\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.456899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.456714 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-model-cache\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.456899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.456830 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-tmp-dir\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.457019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.456959 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-dshm\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.457019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.456999 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-home\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.457128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.457088 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d81db87-2850-47b2-a927-7915d178933f-tls-certs\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.562289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.558102 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-tmp-dir\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.562289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.558220 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-dshm\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.562289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.558287 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-home\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.562289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.558349 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d81db87-2850-47b2-a927-7915d178933f-tls-certs\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.562289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.558439 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l56cn\" (UniqueName: \"kubernetes.io/projected/4d81db87-2850-47b2-a927-7915d178933f-kube-api-access-l56cn\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.562289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.558477 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-model-cache\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.562289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.559027 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-model-cache\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.562289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.559349 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-tmp-dir\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.562289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.561829 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-home\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.571617 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.567215 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-dshm\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.575711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.572240 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d81db87-2850-47b2-a927-7915d178933f-tls-certs\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.596293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.594581 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l56cn\" (UniqueName: \"kubernetes.io/projected/4d81db87-2850-47b2-a927-7915d178933f-kube-api-access-l56cn\") pod \"autores-model-1e0b31-kserve-fd9fb59f7-kb9bg\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.648049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.646595 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:06.986131 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:06.986105 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg"] Oct 30 18:33:06.987803 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:33:06.987775 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d81db87_2850_47b2_a927_7915d178933f.slice/crio-9309d20e5a427be14f18b597e0fa9e10ad50dad1f58721929dc954ba6e6e93c6 WatchSource:0}: Error finding container 9309d20e5a427be14f18b597e0fa9e10ad50dad1f58721929dc954ba6e6e93c6: Status 404 returned error can't find the container with id 9309d20e5a427be14f18b597e0fa9e10ad50dad1f58721929dc954ba6e6e93c6 Oct 30 18:33:07.960023 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:07.959985 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" event={"ID":"4d81db87-2850-47b2-a927-7915d178933f","Type":"ContainerStarted","Data":"aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576"} Oct 30 18:33:07.960023 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:07.960029 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" event={"ID":"4d81db87-2850-47b2-a927-7915d178933f","Type":"ContainerStarted","Data":"9309d20e5a427be14f18b597e0fa9e10ad50dad1f58721929dc954ba6e6e93c6"} Oct 30 18:33:08.000361 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:08.000321 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" podStartSLOduration=2.000307713 podStartE2EDuration="2.000307713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:07.998164319 +0000 UTC m=+2087.047209233" watchObservedRunningTime="2026-10-30 18:33:08.000307713 +0000 UTC m=+2087.049352637" Oct 30 18:33:08.648032 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:08.647994 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:11.644761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:11.644726 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:33:16.647641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:16.647605 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:16.660564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:16.660538 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:17.029659 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:17.029585 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:19.871703 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:33:19.871661 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:33:21.491668 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:21.491642 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:33:21.496490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:21.496467 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:33:30.742827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.742787 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7"] Oct 30 18:33:30.752858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.752833 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.767847 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.767818 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a7508\"/\"openshift-service-ca.crt\"" Oct 30 18:33:30.771534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.771505 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7"] Oct 30 18:33:30.778520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.778485 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a7508\"/\"kube-root-ca.crt\"" Oct 30 18:33:30.778520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.778498 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a7508\"/\"auth-test-model-4a7508-kserve-self-signed-certs\"" Oct 30 18:33:30.778660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.778581 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4a7508\"/\"default-dockercfg-82xwr\"" Oct 30 18:33:30.816025 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.815988 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-dshm\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.816161 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.816116 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-model-cache\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.816226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.816202 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6cwb\" (UniqueName: \"kubernetes.io/projected/2bf6da91-8216-4983-81e9-70b29346c6e9-kube-api-access-d6cwb\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.816384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.816362 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-home\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.816447 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.816432 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf6da91-8216-4983-81e9-70b29346c6e9-tls-certs\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.816599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.816577 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-tmp-dir\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.917864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.917826 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6cwb\" (UniqueName: \"kubernetes.io/projected/2bf6da91-8216-4983-81e9-70b29346c6e9-kube-api-access-d6cwb\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.918498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.918471 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-home\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.919166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.919149 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf6da91-8216-4983-81e9-70b29346c6e9-tls-certs\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.919384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.919365 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-tmp-dir\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.919603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.919587 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-dshm\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.920648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.920630 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-model-cache\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.921636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.920319 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-tmp-dir\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.921806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.919048 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-home\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.921910 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.921039 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-model-cache\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.927866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.927834 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf6da91-8216-4983-81e9-70b29346c6e9-tls-certs\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.932522 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.932496 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-dshm\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:30.936975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:30.936952 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6cwb\" (UniqueName: \"kubernetes.io/projected/2bf6da91-8216-4983-81e9-70b29346c6e9-kube-api-access-d6cwb\") pod \"auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:31.080543 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:31.080508 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:31.495179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:31.495147 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7"] Oct 30 18:33:31.496617 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:33:31.496581 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bf6da91_8216_4983_81e9_70b29346c6e9.slice/crio-0308f07f3fa80600758327fb7f4535d033b6b7c31eb738b13f6634d26d996673 WatchSource:0}: Error finding container 0308f07f3fa80600758327fb7f4535d033b6b7c31eb738b13f6634d26d996673: Status 404 returned error can't find the container with id 0308f07f3fa80600758327fb7f4535d033b6b7c31eb738b13f6634d26d996673 Oct 30 18:33:32.168874 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:32.168835 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" event={"ID":"2bf6da91-8216-4983-81e9-70b29346c6e9","Type":"ContainerStarted","Data":"cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688"} Oct 30 18:33:32.169395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:32.168888 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" event={"ID":"2bf6da91-8216-4983-81e9-70b29346c6e9","Type":"ContainerStarted","Data":"0308f07f3fa80600758327fb7f4535d033b6b7c31eb738b13f6634d26d996673"} Oct 30 18:33:32.209597 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:32.209547 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" podStartSLOduration=2.209534524 podStartE2EDuration="2.209534524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:32.207638313 +0000 UTC m=+2111.256683237" watchObservedRunningTime="2026-10-30 18:33:32.209534524 +0000 UTC m=+2111.258579436" Oct 30 18:33:33.081095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:33.081068 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:33.988661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:33.988630 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm"] Oct 30 18:33:33.992849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:33.992834 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.008842 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.008819 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-ab244c-openshift-default-dockercfg-k9mvg\"" Oct 30 18:33:34.056316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.024984 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm"] Oct 30 18:33:34.112919 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.112893 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istiod-ca-cert\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.113084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.112924 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh9qd\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-kube-api-access-jh9qd\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.113084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.112955 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-envoy\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.113084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.113055 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.113201 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.113104 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-podinfo\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.113338 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.113315 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-token\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.113396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.113356 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-data\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.113447 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.113435 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-credential-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.113479 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.113458 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-certs\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.214954 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.214920 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215143 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.214989 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-podinfo\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215149 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-token\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215187 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-data\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215244 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-credential-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215311 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-certs\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215408 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istiod-ca-cert\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215433 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh9qd\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-kube-api-access-jh9qd\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215435 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215462 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-envoy\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.215652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215582 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-data\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.216065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215893 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-certs\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.216065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.215988 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-credential-socket\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.216428 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.216405 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istiod-ca-cert\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.217776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.217750 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-envoy\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.217882 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.217833 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-podinfo\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.247150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.246385 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-token\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.254183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.254157 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh9qd\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-kube-api-access-jh9qd\") pod \"e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.307021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.306999 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:34.662535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.662509 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm"] Oct 30 18:33:34.663120 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:33:34.663097 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f3e3b7_d1d3_49fb_8106_a26764c54660.slice/crio-6a21c1ef6c246d8890f1417d018d132d9086b483f5801e404e37925ab8274638 WatchSource:0}: Error finding container 6a21c1ef6c246d8890f1417d018d132d9086b483f5801e404e37925ab8274638: Status 404 returned error can't find the container with id 6a21c1ef6c246d8890f1417d018d132d9086b483f5801e404e37925ab8274638 Oct 30 18:33:34.665837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.665797 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:34.665946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.665879 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:34.665946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:34.665926 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:35.223288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:35.220333 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" event={"ID":"a0f3e3b7-d1d3-49fb-8106-a26764c54660","Type":"ContainerStarted","Data":"764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b"} Oct 30 18:33:35.223288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:35.220404 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" event={"ID":"a0f3e3b7-d1d3-49fb-8106-a26764c54660","Type":"ContainerStarted","Data":"6a21c1ef6c246d8890f1417d018d132d9086b483f5801e404e37925ab8274638"} Oct 30 18:33:35.307487 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:35.307458 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:35.308824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:35.308779 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:33:35.308952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:35.308846 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 18:33:36.308157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:36.308120 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:33:36.308804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:36.308769 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 18:33:37.307664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:37.307627 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:33:37.307846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:37.307684 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 18:33:38.315299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:38.313569 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:33:38.315299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:38.313684 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 18:33:39.308069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:39.308036 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:33:39.308232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:39.308108 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 18:33:40.308019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:40.307991 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 18:33:40.308445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:40.308038 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 18:33:40.390322 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:40.371478 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" podStartSLOduration=7.371457654 podStartE2EDuration="7.371457654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:35.296229564 +0000 UTC m=+2114.345274477" watchObservedRunningTime="2026-10-30 18:33:40.371457654 +0000 UTC m=+2119.420502570" Oct 30 18:33:40.395469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:40.395424 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg"] Oct 30 18:33:41.081835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:41.081802 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:41.101747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:41.101717 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:41.303383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:41.303353 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:33:41.497615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:41.497537 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:41.498740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:41.498698 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:41.500386 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:41.500364 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:33:42.288778 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:42.288743 4862 generic.go:356] "Generic (PLEG): container finished" podID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerID="b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d" exitCode=0 Oct 30 18:33:42.288982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:42.288820 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" event={"ID":"a4a7ed92-3470-4734-9896-3ce0def49be1","Type":"ContainerDied","Data":"b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d"} Oct 30 18:33:42.289424 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:42.289398 4862 scope.go:122] "RemoveContainer" containerID="b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d" Oct 30 18:33:42.290792 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:42.290768 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:42.290863 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:42.290831 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:42.290908 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:42.290863 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:43.298008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:43.297972 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" event={"ID":"a4a7ed92-3470-4734-9896-3ce0def49be1","Type":"ContainerStarted","Data":"df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796"} Oct 30 18:33:43.656435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:43.654618 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:33:43.657925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:43.657878 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:33:43.658206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:43.658180 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:33:44.004809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.004727 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk"] Oct 30 18:33:44.007533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.007507 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.023777 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.023748 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-1a2436-openshift-default-dockercfg-qrht6\"" Oct 30 18:33:44.045285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.042157 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk"] Oct 30 18:33:44.193284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.188517 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-podinfo\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.193284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.188616 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-data\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.193284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.188653 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-credential-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.193284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.188708 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istiod-ca-cert\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.193284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.188738 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-certs\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.193284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.188830 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.193284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.188853 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjxt\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-kube-api-access-5fjxt\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.193284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.188901 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-envoy\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.193284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.188964 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-token\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.289979 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-credential-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290063 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istiod-ca-cert\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290093 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-certs\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290175 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290456 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-credential-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290510 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjxt\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-kube-api-access-5fjxt\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290549 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-certs\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290577 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-envoy\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290583 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-socket\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290665 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-token\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290774 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-podinfo\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290792 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istiod-ca-cert\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.290953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.290856 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-data\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.291166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.291146 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-data\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.293454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.293432 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-envoy\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.293903 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.293884 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-podinfo\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.311974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.311946 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-token\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.312257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.312074 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjxt\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-kube-api-access-5fjxt\") pod \"e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.320794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.320775 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:44.654287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.653481 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:33:44.654287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.653537 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:33:44.676447 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.676423 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk"] Oct 30 18:33:44.678043 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:33:44.678017 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd56d44_f570_45f5_9e1a_c3de0ff49b11.slice/crio-05b1256a87f2e94f73066dc6e4391d7d428841a2a1699815874218d4dec71720 WatchSource:0}: Error finding container 05b1256a87f2e94f73066dc6e4391d7d428841a2a1699815874218d4dec71720: Status 404 returned error can't find the container with id 05b1256a87f2e94f73066dc6e4391d7d428841a2a1699815874218d4dec71720 Oct 30 18:33:44.683305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.681400 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:44.683305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.681511 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:44.683305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:44.681565 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:33:45.312819 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:45.312785 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" event={"ID":"9fd56d44-f570-45f5-9e1a-c3de0ff49b11","Type":"ContainerStarted","Data":"4da1817d2f86a529f3eda24b4a9f039ce57c626d4127befa855052d1f1b165b0"} Oct 30 18:33:45.313229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:45.312837 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" event={"ID":"9fd56d44-f570-45f5-9e1a-c3de0ff49b11","Type":"ContainerStarted","Data":"05b1256a87f2e94f73066dc6e4391d7d428841a2a1699815874218d4dec71720"} Oct 30 18:33:45.321207 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:45.321184 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:45.322919 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:45.322892 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:33:45.323033 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:45.322938 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:33:45.687340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:45.686405 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:33:45.691283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:45.687727 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:33:45.691283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:45.687779 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:33:46.321568 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:46.321541 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:33:46.321953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:46.321586 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:33:46.658336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:46.657761 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:33:46.658336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:46.657838 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:33:47.322063 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:47.322035 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:33:47.322483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:47.322081 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:33:47.653804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:47.653708 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:33:47.654195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:47.654148 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:33:48.295690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:48.295623 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" podStartSLOduration=5.295602599 podStartE2EDuration="5.295602599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:45.375838675 +0000 UTC m=+2124.424883589" watchObservedRunningTime="2026-10-30 18:33:48.295602599 +0000 UTC m=+2127.344647513" Oct 30 18:33:48.307116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:48.307075 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:33:48.321146 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:48.321112 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:33:48.321325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:48.321166 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:33:48.659424 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:48.654637 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 18:33:48.659424 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:48.654697 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 18:33:49.321582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:49.321547 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:33:49.321782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:49.321608 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:33:49.660432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:49.660338 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:33:49.661899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:49.661875 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 18:33:49.916256 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:33:49.911709 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:33:50.322064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:50.322020 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:33:50.322256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:50.322095 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:33:51.321824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:51.321781 4862 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:33:51.322384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:51.321835 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:33:52.325495 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:52.325469 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:52.326056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:52.325943 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:52.326627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:52.326608 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:33:53.227797 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:53.227766 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs"] Oct 30 18:33:53.227989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:53.227951 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" containerID="cri-o://60f8c510e1335eca79e9b90e3f7abded45db2f44d5df41803245b53f4bdf91f5" gracePeriod=30 Oct 30 18:33:55.424053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.424017 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" podUID="4d81db87-2850-47b2-a927-7915d178933f" containerName="main" containerID="cri-o://aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576" gracePeriod=45 Oct 30 18:33:55.696526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.696506 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:55.746727 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.746699 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d81db87-2850-47b2-a927-7915d178933f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d81db87-2850-47b2-a927-7915d178933f-tls-certs\") pod \"4d81db87-2850-47b2-a927-7915d178933f\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " Oct 30 18:33:55.746893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.746843 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d81db87-2850-47b2-a927-7915d178933f-kube-api-access-l56cn\" (UniqueName: \"kubernetes.io/projected/4d81db87-2850-47b2-a927-7915d178933f-kube-api-access-l56cn\") pod \"4d81db87-2850-47b2-a927-7915d178933f\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " Oct 30 18:33:55.746893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.746869 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-home\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-home\") pod \"4d81db87-2850-47b2-a927-7915d178933f\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " Oct 30 18:33:55.747022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.746916 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-tmp-dir\") pod \"4d81db87-2850-47b2-a927-7915d178933f\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " Oct 30 18:33:55.747022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.746981 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-model-cache\") pod \"4d81db87-2850-47b2-a927-7915d178933f\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " Oct 30 18:33:55.747022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.747006 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-dshm\") pod \"4d81db87-2850-47b2-a927-7915d178933f\" (UID: \"4d81db87-2850-47b2-a927-7915d178933f\") " Oct 30 18:33:55.747390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.747319 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-tmp-dir" pod "4d81db87-2850-47b2-a927-7915d178933f" (UID: "4d81db87-2850-47b2-a927-7915d178933f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.747510 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.747410 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-home" pod "4d81db87-2850-47b2-a927-7915d178933f" (UID: "4d81db87-2850-47b2-a927-7915d178933f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.747805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.747619 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-model-cache" pod "4d81db87-2850-47b2-a927-7915d178933f" (UID: "4d81db87-2850-47b2-a927-7915d178933f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.747805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.747753 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.747805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.747771 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.747805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.747786 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.749525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.749501 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-dshm" pod "4d81db87-2850-47b2-a927-7915d178933f" (UID: "4d81db87-2850-47b2-a927-7915d178933f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.749688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.749660 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d81db87-2850-47b2-a927-7915d178933f-tls-certs" pod "4d81db87-2850-47b2-a927-7915d178933f" (UID: "4d81db87-2850-47b2-a927-7915d178933f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:55.749793 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.749744 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d81db87-2850-47b2-a927-7915d178933f-kube-api-access-l56cn" pod "4d81db87-2850-47b2-a927-7915d178933f" (UID: "4d81db87-2850-47b2-a927-7915d178933f"). InnerVolumeSpecName "kube-api-access-l56cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:55.849257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.849231 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d81db87-2850-47b2-a927-7915d178933f-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.849257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.849254 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d81db87-2850-47b2-a927-7915d178933f-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.849257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:55.849281 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l56cn\" (UniqueName: \"kubernetes.io/projected/4d81db87-2850-47b2-a927-7915d178933f-kube-api-access-l56cn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:56.419901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:56.419850 4862 generic.go:356] "Generic (PLEG): container finished" podID="4d81db87-2850-47b2-a927-7915d178933f" containerID="aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576" exitCode=0 Oct 30 18:33:56.420076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:56.419933 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" Oct 30 18:33:56.420076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:56.419930 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" event={"ID":"4d81db87-2850-47b2-a927-7915d178933f","Type":"ContainerDied","Data":"aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576"} Oct 30 18:33:56.420076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:56.420042 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg" event={"ID":"4d81db87-2850-47b2-a927-7915d178933f","Type":"ContainerDied","Data":"9309d20e5a427be14f18b597e0fa9e10ad50dad1f58721929dc954ba6e6e93c6"} Oct 30 18:33:56.420076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:56.420060 4862 scope.go:122] "RemoveContainer" containerID="aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576" Oct 30 18:33:56.435011 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:56.434826 4862 scope.go:122] "RemoveContainer" containerID="aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576" Oct 30 18:33:56.435301 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:33:56.435093 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576\": container with ID starting with aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576 not found: ID does not exist" containerID="aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576" Oct 30 18:33:56.435301 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:56.435117 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576"} err="failed to get container status \"aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576\": rpc error: code = NotFound desc = could not find container \"aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576\": container with ID starting with aeffaeea08f86a19da2ae9c54454a9360a3fb0e14f3a52c439586971c47a2576 not found: ID does not exist" Oct 30 18:33:56.470449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:56.470420 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg"] Oct 30 18:33:56.482588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:56.482568 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-1e0b31/autores-model-1e0b31-kserve-fd9fb59f7-kb9bg"] Oct 30 18:33:57.062942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:57.062903 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d81db87-2850-47b2-a927-7915d178933f" path="/var/lib/kubelet/pods/4d81db87-2850-47b2-a927-7915d178933f/volumes" Oct 30 18:33:58.437980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.437879 4862 generic.go:356] "Generic (PLEG): container finished" podID="64350f03-8872-451e-80ed-2f2a7b837c55" containerID="60f8c510e1335eca79e9b90e3f7abded45db2f44d5df41803245b53f4bdf91f5" exitCode=0 Oct 30 18:33:58.437980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.437934 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" event={"ID":"64350f03-8872-451e-80ed-2f2a7b837c55","Type":"ContainerDied","Data":"60f8c510e1335eca79e9b90e3f7abded45db2f44d5df41803245b53f4bdf91f5"} Oct 30 18:33:58.505584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.505525 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:33:58.596745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.596697 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-istio-token\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-istio-token\") pod \"64350f03-8872-451e-80ed-2f2a7b837c55\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " Oct 30 18:33:58.596745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.596760 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-kube-api-access-bvp72\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-kube-api-access-bvp72\") pod \"64350f03-8872-451e-80ed-2f2a7b837c55\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " Oct 30 18:33:58.596745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.596792 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-envoy\") pod \"64350f03-8872-451e-80ed-2f2a7b837c55\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " Oct 30 18:33:58.597109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.596825 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64350f03-8872-451e-80ed-2f2a7b837c55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64350f03-8872-451e-80ed-2f2a7b837c55-istiod-ca-cert\") pod \"64350f03-8872-451e-80ed-2f2a7b837c55\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " Oct 30 18:33:58.597109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.596898 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-certs\") pod \"64350f03-8872-451e-80ed-2f2a7b837c55\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " Oct 30 18:33:58.597240 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.597138 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-socket\") pod \"64350f03-8872-451e-80ed-2f2a7b837c55\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " Oct 30 18:33:58.597240 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.597182 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-credential-socket\") pod \"64350f03-8872-451e-80ed-2f2a7b837c55\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " Oct 30 18:33:58.597520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.597471 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-socket" pod "64350f03-8872-451e-80ed-2f2a7b837c55" (UID: "64350f03-8872-451e-80ed-2f2a7b837c55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:58.597520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.597506 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-credential-socket" pod "64350f03-8872-451e-80ed-2f2a7b837c55" (UID: "64350f03-8872-451e-80ed-2f2a7b837c55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:58.597714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.597554 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64350f03-8872-451e-80ed-2f2a7b837c55-istiod-ca-cert" pod "64350f03-8872-451e-80ed-2f2a7b837c55" (UID: "64350f03-8872-451e-80ed-2f2a7b837c55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:58.597714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.597699 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64350f03-8872-451e-80ed-2f2a7b837c55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64350f03-8872-451e-80ed-2f2a7b837c55-istio-podinfo\") pod \"64350f03-8872-451e-80ed-2f2a7b837c55\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " Oct 30 18:33:58.597849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.597829 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-data\") pod \"64350f03-8872-451e-80ed-2f2a7b837c55\" (UID: \"64350f03-8872-451e-80ed-2f2a7b837c55\") " Oct 30 18:33:58.598552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.598528 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:58.598632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.598558 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:58.598632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.598602 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64350f03-8872-451e-80ed-2f2a7b837c55-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:58.599131 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.599108 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-data" pod "64350f03-8872-451e-80ed-2f2a7b837c55" (UID: "64350f03-8872-451e-80ed-2f2a7b837c55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:58.599469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.599446 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-certs" pod "64350f03-8872-451e-80ed-2f2a7b837c55" (UID: "64350f03-8872-451e-80ed-2f2a7b837c55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:58.599949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.599923 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-envoy" pod "64350f03-8872-451e-80ed-2f2a7b837c55" (UID: "64350f03-8872-451e-80ed-2f2a7b837c55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:58.601200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.601176 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64350f03-8872-451e-80ed-2f2a7b837c55-istio-podinfo" pod "64350f03-8872-451e-80ed-2f2a7b837c55" (UID: "64350f03-8872-451e-80ed-2f2a7b837c55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:58.601408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.601370 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-kube-api-access-bvp72" pod "64350f03-8872-451e-80ed-2f2a7b837c55" (UID: "64350f03-8872-451e-80ed-2f2a7b837c55"). InnerVolumeSpecName "kube-api-access-bvp72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:58.601525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.601452 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-istio-token" pod "64350f03-8872-451e-80ed-2f2a7b837c55" (UID: "64350f03-8872-451e-80ed-2f2a7b837c55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:58.699755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.699727 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvp72\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-kube-api-access-bvp72\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:58.699755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.699755 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:58.699986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.699769 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:58.699986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.699780 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64350f03-8872-451e-80ed-2f2a7b837c55-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:58.699986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.699791 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64350f03-8872-451e-80ed-2f2a7b837c55-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:58.699986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:58.699802 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64350f03-8872-451e-80ed-2f2a7b837c55-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:33:59.445952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:59.445884 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" Oct 30 18:33:59.446366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:59.445876 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs" event={"ID":"64350f03-8872-451e-80ed-2f2a7b837c55","Type":"ContainerDied","Data":"a6ca121a0971c33961624cf2fc24e5cee694c38a8bbd4167e27041eb6ffaea0d"} Oct 30 18:33:59.446366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:59.446018 4862 scope.go:122] "RemoveContainer" containerID="60f8c510e1335eca79e9b90e3f7abded45db2f44d5df41803245b53f4bdf91f5" Oct 30 18:33:59.511249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:59.511220 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs"] Oct 30 18:33:59.522635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:33:59.522606 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-w8sfs"] Oct 30 18:34:01.062175 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:01.062143 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" path="/var/lib/kubelet/pods/64350f03-8872-451e-80ed-2f2a7b837c55/volumes" Oct 30 18:34:06.652846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.652813 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm"] Oct 30 18:34:06.653776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.653755 4862 cpu_manager.go:414] "containerMap: removing container" podUID="4d81db87-2850-47b2-a927-7915d178933f" containerName="main" Oct 30 18:34:06.653845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.653780 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d81db87-2850-47b2-a927-7915d178933f" containerName="main" Oct 30 18:34:06.653845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.653798 4862 cpu_manager.go:414] "containerMap: removing container" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" Oct 30 18:34:06.653845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.653805 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" Oct 30 18:34:06.653973 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.653961 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d81db87-2850-47b2-a927-7915d178933f" containerName="main" Oct 30 18:34:06.654008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.653977 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="64350f03-8872-451e-80ed-2f2a7b837c55" containerName="istio-proxy" Oct 30 18:34:06.659036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.659018 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.686204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.686141 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e95c51\"/\"openshift-service-ca.crt\"" Oct 30 18:34:06.686204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.686155 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e95c51\"/\"kube-root-ca.crt\"" Oct 30 18:34:06.686493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.686214 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e95c51\"/\"default-dockercfg-dpv4s\"" Oct 30 18:34:06.686493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.686448 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e95c51\"/\"auth-test-model-e95c51-kserve-self-signed-certs\"" Oct 30 18:34:06.691280 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.691217 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm"] Oct 30 18:34:06.813741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.813710 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-model-cache\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.813935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.813767 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-dshm\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.813935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.813894 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-tmp-dir\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.814037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.813936 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-home\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.814037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.813964 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629b23f2-dddb-4704-a689-9392ac6942a9-tls-certs\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.814308 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.814279 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6kg\" (UniqueName: \"kubernetes.io/projected/629b23f2-dddb-4704-a689-9392ac6942a9-kube-api-access-2c6kg\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.918385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.917116 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-tmp-dir\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.918385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.917196 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-home\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.918385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.917227 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629b23f2-dddb-4704-a689-9392ac6942a9-tls-certs\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.918385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.917467 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6kg\" (UniqueName: \"kubernetes.io/projected/629b23f2-dddb-4704-a689-9392ac6942a9-kube-api-access-2c6kg\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.918385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.917516 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-model-cache\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.918385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.917548 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-dshm\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.918899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.918873 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-model-cache\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.919170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.919144 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-tmp-dir\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.919469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.919449 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-home\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.946194 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.943975 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-dshm\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.953429 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.947312 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6kg\" (UniqueName: \"kubernetes.io/projected/629b23f2-dddb-4704-a689-9392ac6942a9-kube-api-access-2c6kg\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.974773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.957842 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629b23f2-dddb-4704-a689-9392ac6942a9-tls-certs\") pod \"auth-test-model-e95c51-kserve-7ff4994d49-d5rcm\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:06.981311 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:06.980644 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:07.414308 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:07.407566 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm"] Oct 30 18:34:07.418295 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:34:07.418243 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod629b23f2_dddb_4704_a689_9392ac6942a9.slice/crio-7573d99782be270c3af4ff1300c2753f7f9d6158b7cf5cd555ad4b26e40ac151 WatchSource:0}: Error finding container 7573d99782be270c3af4ff1300c2753f7f9d6158b7cf5cd555ad4b26e40ac151: Status 404 returned error can't find the container with id 7573d99782be270c3af4ff1300c2753f7f9d6158b7cf5cd555ad4b26e40ac151 Oct 30 18:34:07.529645 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:07.529570 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" event={"ID":"629b23f2-dddb-4704-a689-9392ac6942a9","Type":"ContainerStarted","Data":"7573d99782be270c3af4ff1300c2753f7f9d6158b7cf5cd555ad4b26e40ac151"} Oct 30 18:34:08.538327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:08.538293 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" event={"ID":"629b23f2-dddb-4704-a689-9392ac6942a9","Type":"ContainerStarted","Data":"7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842"} Oct 30 18:34:08.585013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:08.584966 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" podStartSLOduration=2.584953178 podStartE2EDuration="2.584953178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:08.583972077 +0000 UTC m=+2147.633016989" watchObservedRunningTime="2026-10-30 18:34:08.584953178 +0000 UTC m=+2147.633998169" Oct 30 18:34:08.981733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:08.981702 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:16.981219 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:16.981186 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:16.993860 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:16.993831 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:17.619523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:17.619498 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:34:18.697793 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.697761 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj"] Oct 30 18:34:18.702098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.702078 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.716466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.716443 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4a7508-openshift-default-dockercfg-ncnp4\"" Oct 30 18:34:18.732180 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.732158 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj"] Oct 30 18:34:18.845030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.845003 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-podinfo\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.845162 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.845043 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-certs\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.845162 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.845074 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.845241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.845196 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6d3ce-dafd-4128-8805-382270e9f63f-istiod-ca-cert\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.845241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.845228 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-token\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.845627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.845343 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-data\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.845627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.845397 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-credential-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.845627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.845419 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz9k5\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-kube-api-access-mz9k5\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.845627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.845457 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-envoy\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.946892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.946865 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-data\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.946903 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-credential-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.946923 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9k5\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-kube-api-access-mz9k5\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.946959 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-envoy\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947007 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-podinfo\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947036 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-certs\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947073 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947113 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6d3ce-dafd-4128-8805-382270e9f63f-istiod-ca-cert\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947140 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-token\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947400 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-data\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947529 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947508 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-credential-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947556 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-socket\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947771 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947711 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-certs\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.947978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.947816 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6d3ce-dafd-4128-8805-382270e9f63f-istiod-ca-cert\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.949427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.949405 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-envoy\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.949676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.949657 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-podinfo\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.965795 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.965764 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-token\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:18.965899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:18.965884 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9k5\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-kube-api-access-mz9k5\") pod \"e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:19.018740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:19.018705 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:19.378729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:19.378699 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj"] Oct 30 18:34:19.379753 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:34:19.379731 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d6d3ce_dafd_4128_8805_382270e9f63f.slice/crio-053a03caa93d4a403f0a198c26d7fd0022c409dafb99f459d5aca1110e55815e WatchSource:0}: Error finding container 053a03caa93d4a403f0a198c26d7fd0022c409dafb99f459d5aca1110e55815e: Status 404 returned error can't find the container with id 053a03caa93d4a403f0a198c26d7fd0022c409dafb99f459d5aca1110e55815e Oct 30 18:34:19.382402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:19.382367 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:34:19.382517 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:19.382449 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:34:19.382517 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:19.382494 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:34:19.626241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:19.626206 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" event={"ID":"30d6d3ce-dafd-4128-8805-382270e9f63f","Type":"ContainerStarted","Data":"40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a"} Oct 30 18:34:19.626445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:19.626271 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" event={"ID":"30d6d3ce-dafd-4128-8805-382270e9f63f","Type":"ContainerStarted","Data":"053a03caa93d4a403f0a198c26d7fd0022c409dafb99f459d5aca1110e55815e"} Oct 30 18:34:19.665603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:19.665490 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" podStartSLOduration=1.665477801 podStartE2EDuration="1.665477801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:19.665224341 +0000 UTC m=+2158.714269255" watchObservedRunningTime="2026-10-30 18:34:19.665477801 +0000 UTC m=+2158.714522711" Oct 30 18:34:19.884317 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:34:19.883660 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:34:20.027638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:20.027571 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:20.043285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:20.029744 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:34:20.043285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:20.029801 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:34:21.027469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:21.027439 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:34:21.027850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:21.027488 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:34:22.027216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:22.027141 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:34:22.027216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:22.027197 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:34:23.027593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:23.027561 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:34:23.027972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:23.027616 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:34:24.027463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:24.027430 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:34:24.027632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:24.027485 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:34:25.027150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:25.027119 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 18:34:25.027352 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:25.027177 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 18:34:26.034805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:26.034772 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:26.036193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:26.036169 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:26.040917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:26.040861 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:34:32.460823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:34:32.460786 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:34:49.907970 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:34:49.907928 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:35:19.869794 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:35:19.869741 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:35:24.773656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:24.773619 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:35:27.582173 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.582136 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn"] Oct 30 18:35:27.587325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.587302 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.602548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.602525 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn"] Oct 30 18:35:27.611445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.611422 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-160bf8\"/\"kube-root-ca.crt\"" Oct 30 18:35:27.614003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.613981 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-160bf8\"/\"default-dockercfg-4jh5f\"" Oct 30 18:35:27.614256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.614239 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-160bf8\"/\"openshift-service-ca.crt\"" Oct 30 18:35:27.614451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.614431 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-160bf8\"/\"explicit-model-160bf8-kserve-self-signed-certs\"" Oct 30 18:35:27.650380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.650356 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-home\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.650493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.650411 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tls-certs\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.650534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.650501 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-model-cache\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.650602 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.650589 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tmp-dir\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.650641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.650618 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-dshm\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.650792 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.650775 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhj99\" (UniqueName: \"kubernetes.io/projected/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-kube-api-access-dhj99\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.751766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.751731 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tls-certs\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.751959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.751787 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-model-cache\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.751959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.751859 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tmp-dir\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.751959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.751886 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-dshm\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.752134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.751986 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhj99\" (UniqueName: \"kubernetes.io/projected/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-kube-api-access-dhj99\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.752134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.752042 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-home\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.752388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.752361 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-model-cache\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.752571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.752457 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-home\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.752682 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.752610 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tmp-dir\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.755065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.755038 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-dshm\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.755601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.755580 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tls-certs\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.771758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.771731 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhj99\" (UniqueName: \"kubernetes.io/projected/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-kube-api-access-dhj99\") pod \"explicit-model-160bf8-kserve-77578494b9-2mdfn\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:27.902907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:27.902555 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:28.280461 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:28.280434 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn"] Oct 30 18:35:28.281380 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:35:28.281353 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1f41686_3dbf_4bdb_b8c8_71cadee5fedd.slice/crio-3e5ad7165621f19e9af1f9ae358f2a4d262d864ced4506c15d1835b6f6bdbf11 WatchSource:0}: Error finding container 3e5ad7165621f19e9af1f9ae358f2a4d262d864ced4506c15d1835b6f6bdbf11: Status 404 returned error can't find the container with id 3e5ad7165621f19e9af1f9ae358f2a4d262d864ced4506c15d1835b6f6bdbf11 Oct 30 18:35:29.226983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:29.226942 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" event={"ID":"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd","Type":"ContainerStarted","Data":"19e8db1e4ef0fb6cf32450ab9c536ad6e26104e51cff009d64726ffaaf10d515"} Oct 30 18:35:29.227450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:29.226994 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" event={"ID":"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd","Type":"ContainerStarted","Data":"3e5ad7165621f19e9af1f9ae358f2a4d262d864ced4506c15d1835b6f6bdbf11"} Oct 30 18:35:29.267051 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:29.267005 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" podStartSLOduration=2.266992289 podStartE2EDuration="2.266992289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:29.264733578 +0000 UTC m=+2228.313778491" watchObservedRunningTime="2026-10-30 18:35:29.266992289 +0000 UTC m=+2228.316037201" Oct 30 18:35:29.903220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:29.903185 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:37.904329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:37.904302 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:37.917812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:37.917787 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:38.294644 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:38.294564 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:35:49.870031 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:35:49.869991 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:35:53.255225 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:35:53.255183 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn"] Oct 30 18:36:07.119635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:07.119604 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:36:08.278016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.277970 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" podUID="c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" containerName="main" containerID="cri-o://19e8db1e4ef0fb6cf32450ab9c536ad6e26104e51cff009d64726ffaaf10d515" gracePeriod=45 Oct 30 18:36:08.284063 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.284027 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" podUID="c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": dial tcp 10.133.0.107:8000: connect: connection refused" Oct 30 18:36:08.503141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.503098 4862 generic.go:356] "Generic (PLEG): container finished" podID="c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" containerID="19e8db1e4ef0fb6cf32450ab9c536ad6e26104e51cff009d64726ffaaf10d515" exitCode=0 Oct 30 18:36:08.503402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.503327 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" event={"ID":"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd","Type":"ContainerDied","Data":"19e8db1e4ef0fb6cf32450ab9c536ad6e26104e51cff009d64726ffaaf10d515"} Oct 30 18:36:08.712421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.711589 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:36:08.881293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.878345 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tmp-dir\") pod \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " Oct 30 18:36:08.881293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.878581 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-dshm\") pod \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " Oct 30 18:36:08.881293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.878695 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-home\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-home\") pod \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " Oct 30 18:36:08.881293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.878734 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tls-certs\") pod \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " Oct 30 18:36:08.881293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.878806 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-model-cache\") pod \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " Oct 30 18:36:08.881293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.878833 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-kube-api-access-dhj99\" (UniqueName: \"kubernetes.io/projected/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-kube-api-access-dhj99\") pod \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\" (UID: \"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd\") " Oct 30 18:36:08.881293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.880464 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-model-cache" pod "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" (UID: "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:08.881293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.880644 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tmp-dir" pod "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" (UID: "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:08.881293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.881285 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-home" pod "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" (UID: "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:08.883308 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.882470 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-kube-api-access-dhj99" pod "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" (UID: "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd"). InnerVolumeSpecName "kube-api-access-dhj99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:08.883931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.883910 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-dshm" pod "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" (UID: "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:08.884079 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.884005 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tls-certs" pod "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" (UID: "c1f41686-3dbf-4bdb-b8c8-71cadee5fedd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:08.980351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.980315 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:08.980351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.980351 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:08.980351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.980359 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:08.980351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.980368 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:08.980648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.980377 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:08.980648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:08.980386 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhj99\" (UniqueName: \"kubernetes.io/projected/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd-kube-api-access-dhj99\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:09.511743 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:09.511716 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" Oct 30 18:36:09.512186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:09.511741 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn" event={"ID":"c1f41686-3dbf-4bdb-b8c8-71cadee5fedd","Type":"ContainerDied","Data":"3e5ad7165621f19e9af1f9ae358f2a4d262d864ced4506c15d1835b6f6bdbf11"} Oct 30 18:36:09.512186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:09.511799 4862 scope.go:122] "RemoveContainer" containerID="19e8db1e4ef0fb6cf32450ab9c536ad6e26104e51cff009d64726ffaaf10d515" Oct 30 18:36:09.564395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:09.564366 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn"] Oct 30 18:36:09.574340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:09.574316 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-160bf8/explicit-model-160bf8-kserve-77578494b9-2mdfn"] Oct 30 18:36:11.063006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:11.062975 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" path="/var/lib/kubelet/pods/c1f41686-3dbf-4bdb-b8c8-71cadee5fedd/volumes" Oct 30 18:36:19.873064 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:36:19.873023 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:36:49.873046 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:36:49.872964 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:36:53.654253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:53.654222 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z"] Oct 30 18:36:53.654717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:53.654431 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" containerID="cri-o://0c73a1b1b2084d38ff6f01c04a5bbc275cadfa147b14e51b3cffb1df6fe52a2a" gracePeriod=30 Oct 30 18:36:56.127737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.127705 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs"] Oct 30 18:36:56.128233 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.128220 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" containerName="main" Oct 30 18:36:56.128335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.128234 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" containerName="main" Oct 30 18:36:56.128385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.128366 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f41686-3dbf-4bdb-b8c8-71cadee5fedd" containerName="main" Oct 30 18:36:56.131403 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.131388 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.151060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.151037 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs"] Oct 30 18:36:56.160045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.160023 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:36:56.160303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.160283 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:36:56.189142 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.189116 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fg2\" (UniqueName: \"kubernetes.io/projected/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-kube-api-access-v4fg2\") pod \"maas-api-revoke-sub-6737f14b-zbnxs\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.189371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.189353 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6737f14b-zbnxs\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.189460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.189391 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6737f14b-zbnxs\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.303347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.302283 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6737f14b-zbnxs\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.303347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.302355 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6737f14b-zbnxs\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.303347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.302465 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fg2\" (UniqueName: \"kubernetes.io/projected/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-kube-api-access-v4fg2\") pod \"maas-api-revoke-sub-6737f14b-zbnxs\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.308378 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.308347 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6737f14b-zbnxs\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.312858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.312830 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6737f14b-zbnxs\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.324906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.324874 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fg2\" (UniqueName: \"kubernetes.io/projected/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-kube-api-access-v4fg2\") pod \"maas-api-revoke-sub-6737f14b-zbnxs\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.444987 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.444905 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:56.752362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.752241 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs"] Oct 30 18:36:56.756916 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:36:56.756888 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2504490a_8974_45de_a2b9_dc7ec1b2c6c5.slice/crio-c5094aec455f501ceb4263fcd5a33a0dc6f1d92746612b946337293eafd2dc5b WatchSource:0}: Error finding container c5094aec455f501ceb4263fcd5a33a0dc6f1d92746612b946337293eafd2dc5b: Status 404 returned error can't find the container with id c5094aec455f501ceb4263fcd5a33a0dc6f1d92746612b946337293eafd2dc5b Oct 30 18:36:56.866697 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:56.866667 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" event={"ID":"2504490a-8974-45de-a2b9-dc7ec1b2c6c5","Type":"ContainerStarted","Data":"c5094aec455f501ceb4263fcd5a33a0dc6f1d92746612b946337293eafd2dc5b"} Oct 30 18:36:57.874730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:57.874696 4862 generic.go:356] "Generic (PLEG): container finished" podID="2504490a-8974-45de-a2b9-dc7ec1b2c6c5" containerID="c56a3dab1d7f69a775404f6f97ac3862907ac8be0a1e204f0fed6357426e1749" exitCode=0 Oct 30 18:36:57.875105 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:57.874761 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" event={"ID":"2504490a-8974-45de-a2b9-dc7ec1b2c6c5","Type":"ContainerDied","Data":"c56a3dab1d7f69a775404f6f97ac3862907ac8be0a1e204f0fed6357426e1749"} Oct 30 18:36:58.884496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:58.884463 4862 generic.go:356] "Generic (PLEG): container finished" podID="3d13ba25-0745-413d-8667-3edb42af0102" containerID="0c73a1b1b2084d38ff6f01c04a5bbc275cadfa147b14e51b3cffb1df6fe52a2a" exitCode=0 Oct 30 18:36:58.884869 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:58.884836 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" event={"ID":"3d13ba25-0745-413d-8667-3edb42af0102","Type":"ContainerDied","Data":"0c73a1b1b2084d38ff6f01c04a5bbc275cadfa147b14e51b3cffb1df6fe52a2a"} Oct 30 18:36:58.932867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:58.932806 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:36:59.040374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040346 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-istio-token\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-istio-token\") pod \"3d13ba25-0745-413d-8667-3edb42af0102\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " Oct 30 18:36:59.040508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040409 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-certs\") pod \"3d13ba25-0745-413d-8667-3edb42af0102\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " Oct 30 18:36:59.040508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040434 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-credential-socket\") pod \"3d13ba25-0745-413d-8667-3edb42af0102\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " Oct 30 18:36:59.040590 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040552 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-envoy\") pod \"3d13ba25-0745-413d-8667-3edb42af0102\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " Oct 30 18:36:59.040637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040621 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d13ba25-0745-413d-8667-3edb42af0102-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d13ba25-0745-413d-8667-3edb42af0102-istiod-ca-cert\") pod \"3d13ba25-0745-413d-8667-3edb42af0102\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " Oct 30 18:36:59.040685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040674 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-data\") pod \"3d13ba25-0745-413d-8667-3edb42af0102\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " Oct 30 18:36:59.040741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040715 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-kube-api-access-cttm7\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-kube-api-access-cttm7\") pod \"3d13ba25-0745-413d-8667-3edb42af0102\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " Oct 30 18:36:59.040798 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040729 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-certs" pod "3d13ba25-0745-413d-8667-3edb42af0102" (UID: "3d13ba25-0745-413d-8667-3edb42af0102"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:59.040798 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040747 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-credential-socket" pod "3d13ba25-0745-413d-8667-3edb42af0102" (UID: "3d13ba25-0745-413d-8667-3edb42af0102"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:59.040798 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040764 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-socket\") pod \"3d13ba25-0745-413d-8667-3edb42af0102\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " Oct 30 18:36:59.040798 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040794 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d13ba25-0745-413d-8667-3edb42af0102-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d13ba25-0745-413d-8667-3edb42af0102-istio-podinfo\") pod \"3d13ba25-0745-413d-8667-3edb42af0102\" (UID: \"3d13ba25-0745-413d-8667-3edb42af0102\") " Oct 30 18:36:59.041012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.040946 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-data" pod "3d13ba25-0745-413d-8667-3edb42af0102" (UID: "3d13ba25-0745-413d-8667-3edb42af0102"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:59.041012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.041004 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-socket" pod "3d13ba25-0745-413d-8667-3edb42af0102" (UID: "3d13ba25-0745-413d-8667-3edb42af0102"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:59.041383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.041348 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d13ba25-0745-413d-8667-3edb42af0102-istiod-ca-cert" pod "3d13ba25-0745-413d-8667-3edb42af0102" (UID: "3d13ba25-0745-413d-8667-3edb42af0102"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:59.041696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.041666 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.041935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.041700 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.041935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.041718 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.041935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.041731 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d13ba25-0745-413d-8667-3edb42af0102-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.041935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.041746 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.043044 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.043019 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-envoy" pod "3d13ba25-0745-413d-8667-3edb42af0102" (UID: "3d13ba25-0745-413d-8667-3edb42af0102"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:36:59.043140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.043063 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-kube-api-access-cttm7" pod "3d13ba25-0745-413d-8667-3edb42af0102" (UID: "3d13ba25-0745-413d-8667-3edb42af0102"). InnerVolumeSpecName "kube-api-access-cttm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:59.043183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.043154 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-istio-token" pod "3d13ba25-0745-413d-8667-3edb42af0102" (UID: "3d13ba25-0745-413d-8667-3edb42af0102"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:59.043541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.043526 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d13ba25-0745-413d-8667-3edb42af0102-istio-podinfo" pod "3d13ba25-0745-413d-8667-3edb42af0102" (UID: "3d13ba25-0745-413d-8667-3edb42af0102"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:36:59.070886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.070869 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:59.144362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.143151 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-xks-ca\") pod \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " Oct 30 18:36:59.144362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.143208 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-openshift-ca\") pod \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " Oct 30 18:36:59.144362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.143237 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-kube-api-access-v4fg2\" (UniqueName: \"kubernetes.io/projected/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-kube-api-access-v4fg2\") pod \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\" (UID: \"2504490a-8974-45de-a2b9-dc7ec1b2c6c5\") " Oct 30 18:36:59.144362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.143860 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-openshift-ca" pod "2504490a-8974-45de-a2b9-dc7ec1b2c6c5" (UID: "2504490a-8974-45de-a2b9-dc7ec1b2c6c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:59.144362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.144087 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d13ba25-0745-413d-8667-3edb42af0102-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.144362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.144110 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cttm7\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-kube-api-access-cttm7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.144362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.144125 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d13ba25-0745-413d-8667-3edb42af0102-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.144362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.144138 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d13ba25-0745-413d-8667-3edb42af0102-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.144362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.144153 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.146729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.146689 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-kube-api-access-v4fg2" pod "2504490a-8974-45de-a2b9-dc7ec1b2c6c5" (UID: "2504490a-8974-45de-a2b9-dc7ec1b2c6c5"). InnerVolumeSpecName "kube-api-access-v4fg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:59.146852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.146782 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-xks-ca" pod "2504490a-8974-45de-a2b9-dc7ec1b2c6c5" (UID: "2504490a-8974-45de-a2b9-dc7ec1b2c6c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:59.245761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.245711 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.245761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.245731 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4fg2\" (UniqueName: \"kubernetes.io/projected/2504490a-8974-45de-a2b9-dc7ec1b2c6c5-kube-api-access-v4fg2\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:36:59.892244 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.892218 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" Oct 30 18:36:59.892718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.892257 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z" event={"ID":"3d13ba25-0745-413d-8667-3edb42af0102","Type":"ContainerDied","Data":"4a885bba1ec69325754a253280dffb97a1e29fda3028f5837ed3b943b90b20f7"} Oct 30 18:36:59.892718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.892339 4862 scope.go:122] "RemoveContainer" containerID="0c73a1b1b2084d38ff6f01c04a5bbc275cadfa147b14e51b3cffb1df6fe52a2a" Oct 30 18:36:59.893746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.893717 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" Oct 30 18:36:59.893831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.893717 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs" event={"ID":"2504490a-8974-45de-a2b9-dc7ec1b2c6c5","Type":"ContainerDied","Data":"c5094aec455f501ceb4263fcd5a33a0dc6f1d92746612b946337293eafd2dc5b"} Oct 30 18:36:59.893892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.893837 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5094aec455f501ceb4263fcd5a33a0dc6f1d92746612b946337293eafd2dc5b" Oct 30 18:36:59.970942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.970892 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z"] Oct 30 18:36:59.982728 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:36:59.982706 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-s7c2z"] Oct 30 18:37:01.070850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:01.070822 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d13ba25-0745-413d-8667-3edb42af0102" path="/var/lib/kubelet/pods/3d13ba25-0745-413d-8667-3edb42af0102/volumes" Oct 30 18:37:08.697489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:08.697451 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs"] Oct 30 18:37:08.697908 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:08.697633 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" containerID="cri-o://4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06" gracePeriod=30 Oct 30 18:37:10.684740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:10.684705 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:37:13.965109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.965086 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:37:13.988134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.988104 4862 generic.go:356] "Generic (PLEG): container finished" podID="11730e43-29fb-4f70-b591-afed0868b014" containerID="4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06" exitCode=0 Oct 30 18:37:13.988297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.988143 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" Oct 30 18:37:13.988297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.988154 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" event={"ID":"11730e43-29fb-4f70-b591-afed0868b014","Type":"ContainerDied","Data":"4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06"} Oct 30 18:37:13.988297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.988226 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs" event={"ID":"11730e43-29fb-4f70-b591-afed0868b014","Type":"ContainerDied","Data":"12d9fba9b73c075dc1227757f45199069eec4c2409126662494077cfa9f87bc0"} Oct 30 18:37:13.988297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.988257 4862 scope.go:122] "RemoveContainer" containerID="4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06" Oct 30 18:37:13.992006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.991986 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-credential-socket\") pod \"11730e43-29fb-4f70-b591-afed0868b014\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " Oct 30 18:37:13.992132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992049 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-envoy\") pod \"11730e43-29fb-4f70-b591-afed0868b014\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " Oct 30 18:37:13.992132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992095 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-data\") pod \"11730e43-29fb-4f70-b591-afed0868b014\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " Oct 30 18:37:13.992240 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992191 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-certs\") pod \"11730e43-29fb-4f70-b591-afed0868b014\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " Oct 30 18:37:13.992240 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992233 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-kube-api-access-xp269\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-kube-api-access-xp269\") pod \"11730e43-29fb-4f70-b591-afed0868b014\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " Oct 30 18:37:13.992370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992304 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-istio-token\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-istio-token\") pod \"11730e43-29fb-4f70-b591-afed0868b014\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " Oct 30 18:37:13.992370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992311 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-credential-socket" pod "11730e43-29fb-4f70-b591-afed0868b014" (UID: "11730e43-29fb-4f70-b591-afed0868b014"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:13.992472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992375 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11730e43-29fb-4f70-b591-afed0868b014-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11730e43-29fb-4f70-b591-afed0868b014-istio-podinfo\") pod \"11730e43-29fb-4f70-b591-afed0868b014\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " Oct 30 18:37:13.992472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992432 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11730e43-29fb-4f70-b591-afed0868b014-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11730e43-29fb-4f70-b591-afed0868b014-istiod-ca-cert\") pod \"11730e43-29fb-4f70-b591-afed0868b014\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " Oct 30 18:37:13.992583 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992493 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-socket\") pod \"11730e43-29fb-4f70-b591-afed0868b014\" (UID: \"11730e43-29fb-4f70-b591-afed0868b014\") " Oct 30 18:37:13.992583 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992507 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-data" pod "11730e43-29fb-4f70-b591-afed0868b014" (UID: "11730e43-29fb-4f70-b591-afed0868b014"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:13.992583 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992527 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-certs" pod "11730e43-29fb-4f70-b591-afed0868b014" (UID: "11730e43-29fb-4f70-b591-afed0868b014"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:13.992872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.992852 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11730e43-29fb-4f70-b591-afed0868b014-istiod-ca-cert" pod "11730e43-29fb-4f70-b591-afed0868b014" (UID: "11730e43-29fb-4f70-b591-afed0868b014"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:13.994309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.993057 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-socket" pod "11730e43-29fb-4f70-b591-afed0868b014" (UID: "11730e43-29fb-4f70-b591-afed0868b014"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:13.994309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.993621 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11730e43-29fb-4f70-b591-afed0868b014-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:13.994309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.993643 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:13.994309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.993659 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:13.994309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.993673 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:13.994309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.993689 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:13.994716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.994487 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-envoy" pod "11730e43-29fb-4f70-b591-afed0868b014" (UID: "11730e43-29fb-4f70-b591-afed0868b014"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:13.994892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.994870 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11730e43-29fb-4f70-b591-afed0868b014-istio-podinfo" pod "11730e43-29fb-4f70-b591-afed0868b014" (UID: "11730e43-29fb-4f70-b591-afed0868b014"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:13.995113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.995096 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-kube-api-access-xp269" pod "11730e43-29fb-4f70-b591-afed0868b014" (UID: "11730e43-29fb-4f70-b591-afed0868b014"). InnerVolumeSpecName "kube-api-access-xp269". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:13.995291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.995241 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-istio-token" pod "11730e43-29fb-4f70-b591-afed0868b014" (UID: "11730e43-29fb-4f70-b591-afed0868b014"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:13.999346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.999329 4862 scope.go:122] "RemoveContainer" containerID="4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06" Oct 30 18:37:13.999638 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:37:13.999611 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06\": container with ID starting with 4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06 not found: ID does not exist" containerID="4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06" Oct 30 18:37:13.999708 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:13.999647 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06"} err="failed to get container status \"4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06\": rpc error: code = NotFound desc = could not find container \"4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06\": container with ID starting with 4043fdcd43e78d5640a782632072185dcf9899484c6f2161ade1fc821a4fab06 not found: ID does not exist" Oct 30 18:37:14.094914 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:14.094887 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp269\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-kube-api-access-xp269\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.094914 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:14.094911 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11730e43-29fb-4f70-b591-afed0868b014-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.095093 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:14.094923 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11730e43-29fb-4f70-b591-afed0868b014-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.095093 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:14.094936 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11730e43-29fb-4f70-b591-afed0868b014-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:14.339097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:14.339067 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs"] Oct 30 18:37:14.352204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:14.352166 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-8b9cs"] Oct 30 18:37:15.073815 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:15.073782 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11730e43-29fb-4f70-b591-afed0868b014" path="/var/lib/kubelet/pods/11730e43-29fb-4f70-b591-afed0868b014/volumes" Oct 30 18:37:19.885745 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:37:19.885709 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:37:23.911538 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:23.911506 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf"] Oct 30 18:37:23.911946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:23.911716 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" containerID="cri-o://d18dff880c6ea486ac959cdafb6c73f2ffb7e0ec47b13faa7685b5522f12e531" gracePeriod=30 Oct 30 18:37:23.923522 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:23.923489 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt"] Oct 30 18:37:23.923742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:23.923704 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" containerID="cri-o://8732922daf4c4d1bae6bff36377f44fcf6e4b912eb364d0509778dc63a3eca10" gracePeriod=30 Oct 30 18:37:29.088966 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.088884 4862 generic.go:356] "Generic (PLEG): container finished" podID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerID="d18dff880c6ea486ac959cdafb6c73f2ffb7e0ec47b13faa7685b5522f12e531" exitCode=0 Oct 30 18:37:29.089766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.089088 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" event={"ID":"11d91bf0-629e-4d73-a340-36d38891b3b2","Type":"ContainerDied","Data":"d18dff880c6ea486ac959cdafb6c73f2ffb7e0ec47b13faa7685b5522f12e531"} Oct 30 18:37:29.091332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.091147 4862 generic.go:356] "Generic (PLEG): container finished" podID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerID="8732922daf4c4d1bae6bff36377f44fcf6e4b912eb364d0509778dc63a3eca10" exitCode=0 Oct 30 18:37:29.091332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.091220 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" event={"ID":"45344089-c81d-4ee0-8615-b6ad76d4027d","Type":"ContainerDied","Data":"8732922daf4c4d1bae6bff36377f44fcf6e4b912eb364d0509778dc63a3eca10"} Oct 30 18:37:29.263867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.263844 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:37:29.266874 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.266854 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.423802 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-token\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-token\") pod \"45344089-c81d-4ee0-8615-b6ad76d4027d\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.423880 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-certs\") pod \"11d91bf0-629e-4d73-a340-36d38891b3b2\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.423921 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-podinfo\") pod \"11d91bf0-629e-4d73-a340-36d38891b3b2\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.423981 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-socket\") pod \"45344089-c81d-4ee0-8615-b6ad76d4027d\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424021 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-credential-socket\") pod \"11d91bf0-629e-4d73-a340-36d38891b3b2\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424048 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-podinfo\") pod \"45344089-c81d-4ee0-8615-b6ad76d4027d\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424093 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-envoy\") pod \"45344089-c81d-4ee0-8615-b6ad76d4027d\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424159 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-kube-api-access-cp8v5\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-kube-api-access-cp8v5\") pod \"11d91bf0-629e-4d73-a340-36d38891b3b2\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424213 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-credential-socket\") pod \"45344089-c81d-4ee0-8615-b6ad76d4027d\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424276 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-kube-api-access-d2nrv\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-kube-api-access-d2nrv\") pod \"45344089-c81d-4ee0-8615-b6ad76d4027d\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424314 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-socket\") pod \"11d91bf0-629e-4d73-a340-36d38891b3b2\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424350 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-envoy\") pod \"11d91bf0-629e-4d73-a340-36d38891b3b2\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424379 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45344089-c81d-4ee0-8615-b6ad76d4027d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45344089-c81d-4ee0-8615-b6ad76d4027d-istiod-ca-cert\") pod \"45344089-c81d-4ee0-8615-b6ad76d4027d\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424431 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-token\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-token\") pod \"11d91bf0-629e-4d73-a340-36d38891b3b2\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424495 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d91bf0-629e-4d73-a340-36d38891b3b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d91bf0-629e-4d73-a340-36d38891b3b2-istiod-ca-cert\") pod \"11d91bf0-629e-4d73-a340-36d38891b3b2\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424584 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-certs\") pod \"45344089-c81d-4ee0-8615-b6ad76d4027d\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424617 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-data\") pod \"11d91bf0-629e-4d73-a340-36d38891b3b2\" (UID: \"11d91bf0-629e-4d73-a340-36d38891b3b2\") " Oct 30 18:37:29.425452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.424645 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-data\") pod \"45344089-c81d-4ee0-8615-b6ad76d4027d\" (UID: \"45344089-c81d-4ee0-8615-b6ad76d4027d\") " Oct 30 18:37:29.427073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.426753 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-credential-socket" pod "45344089-c81d-4ee0-8615-b6ad76d4027d" (UID: "45344089-c81d-4ee0-8615-b6ad76d4027d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.427476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.427209 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-data" pod "45344089-c81d-4ee0-8615-b6ad76d4027d" (UID: "45344089-c81d-4ee0-8615-b6ad76d4027d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.431985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.430456 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-credential-socket" pod "11d91bf0-629e-4d73-a340-36d38891b3b2" (UID: "11d91bf0-629e-4d73-a340-36d38891b3b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.431985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.430673 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-certs" pod "11d91bf0-629e-4d73-a340-36d38891b3b2" (UID: "11d91bf0-629e-4d73-a340-36d38891b3b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.432788 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.432757 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-socket" pod "45344089-c81d-4ee0-8615-b6ad76d4027d" (UID: "45344089-c81d-4ee0-8615-b6ad76d4027d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.433223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.433185 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-certs" pod "45344089-c81d-4ee0-8615-b6ad76d4027d" (UID: "45344089-c81d-4ee0-8615-b6ad76d4027d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.433525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.433499 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d91bf0-629e-4d73-a340-36d38891b3b2-istiod-ca-cert" pod "11d91bf0-629e-4d73-a340-36d38891b3b2" (UID: "11d91bf0-629e-4d73-a340-36d38891b3b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:29.434841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.434802 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-data" pod "11d91bf0-629e-4d73-a340-36d38891b3b2" (UID: "11d91bf0-629e-4d73-a340-36d38891b3b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.435012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.434989 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45344089-c81d-4ee0-8615-b6ad76d4027d-istiod-ca-cert" pod "45344089-c81d-4ee0-8615-b6ad76d4027d" (UID: "45344089-c81d-4ee0-8615-b6ad76d4027d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:29.435093 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.435048 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-socket" pod "11d91bf0-629e-4d73-a340-36d38891b3b2" (UID: "11d91bf0-629e-4d73-a340-36d38891b3b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.438493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.435903 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-podinfo" pod "11d91bf0-629e-4d73-a340-36d38891b3b2" (UID: "11d91bf0-629e-4d73-a340-36d38891b3b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:29.438493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.437352 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-kube-api-access-d2nrv" pod "45344089-c81d-4ee0-8615-b6ad76d4027d" (UID: "45344089-c81d-4ee0-8615-b6ad76d4027d"). InnerVolumeSpecName "kube-api-access-d2nrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:29.438493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.437444 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-token" pod "45344089-c81d-4ee0-8615-b6ad76d4027d" (UID: "45344089-c81d-4ee0-8615-b6ad76d4027d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:29.438493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.438237 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-envoy" pod "11d91bf0-629e-4d73-a340-36d38891b3b2" (UID: "11d91bf0-629e-4d73-a340-36d38891b3b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.438493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.438462 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-token" pod "11d91bf0-629e-4d73-a340-36d38891b3b2" (UID: "11d91bf0-629e-4d73-a340-36d38891b3b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:29.439325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.439248 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-kube-api-access-cp8v5" pod "11d91bf0-629e-4d73-a340-36d38891b3b2" (UID: "11d91bf0-629e-4d73-a340-36d38891b3b2"). InnerVolumeSpecName "kube-api-access-cp8v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:29.441879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.441844 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-podinfo" pod "45344089-c81d-4ee0-8615-b6ad76d4027d" (UID: "45344089-c81d-4ee0-8615-b6ad76d4027d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:37:29.460849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.460820 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-envoy" pod "45344089-c81d-4ee0-8615-b6ad76d4027d" (UID: "45344089-c81d-4ee0-8615-b6ad76d4027d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:29.526880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526810 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.526880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526838 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.526880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526847 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.526880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526854 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.526880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526862 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.526880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526869 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.526880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526877 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526885 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526895 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526905 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526914 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp8v5\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-kube-api-access-cp8v5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526924 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45344089-c81d-4ee0-8615-b6ad76d4027d-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526932 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2nrv\" (UniqueName: \"kubernetes.io/projected/45344089-c81d-4ee0-8615-b6ad76d4027d-kube-api-access-d2nrv\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526940 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526948 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526955 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45344089-c81d-4ee0-8615-b6ad76d4027d-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526963 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d91bf0-629e-4d73-a340-36d38891b3b2-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:29.527205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:29.526972 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d91bf0-629e-4d73-a340-36d38891b3b2-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:37:30.099233 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.099200 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" Oct 30 18:37:30.099688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.099229 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf" event={"ID":"11d91bf0-629e-4d73-a340-36d38891b3b2","Type":"ContainerDied","Data":"f11d8235f24bde87d1bf94a584943024ea5370f2e844cbd3effe7ab8b2a88f6d"} Oct 30 18:37:30.099688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.099322 4862 scope.go:122] "RemoveContainer" containerID="d18dff880c6ea486ac959cdafb6c73f2ffb7e0ec47b13faa7685b5522f12e531" Oct 30 18:37:30.100908 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.100882 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" event={"ID":"45344089-c81d-4ee0-8615-b6ad76d4027d","Type":"ContainerDied","Data":"6452b078b9444b27b8988925f374040302a2840780a207b1c0ee4bcedc21303c"} Oct 30 18:37:30.101017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.100934 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt" Oct 30 18:37:30.109275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.109243 4862 scope.go:122] "RemoveContainer" containerID="8732922daf4c4d1bae6bff36377f44fcf6e4b912eb364d0509778dc63a3eca10" Oct 30 18:37:30.164205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.164171 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt"] Oct 30 18:37:30.176292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.176273 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fe1374-openshift-default-5dc59487b5-pjcpt"] Oct 30 18:37:30.204586 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.204532 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf"] Oct 30 18:37:30.208426 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:30.208403 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tlvmf"] Oct 30 18:37:31.062609 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:31.062579 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" path="/var/lib/kubelet/pods/11d91bf0-629e-4d73-a340-36d38891b3b2/volumes" Oct 30 18:37:31.063013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:31.062999 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" path="/var/lib/kubelet/pods/45344089-c81d-4ee0-8615-b6ad76d4027d/volumes" Oct 30 18:37:44.738914 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:44.738880 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7"] Oct 30 18:37:49.869725 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:37:49.869684 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:37:59.764062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:37:59.764018 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" podUID="2bf6da91-8216-4983-81e9-70b29346c6e9" containerName="main" containerID="cri-o://cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688" gracePeriod=45 Oct 30 18:38:00.069716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.069695 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:38:00.171332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.171304 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-model-cache\") pod \"2bf6da91-8216-4983-81e9-70b29346c6e9\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " Oct 30 18:38:00.171512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.171336 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-home\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-home\") pod \"2bf6da91-8216-4983-81e9-70b29346c6e9\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " Oct 30 18:38:00.171512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.171423 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-tmp-dir\") pod \"2bf6da91-8216-4983-81e9-70b29346c6e9\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " Oct 30 18:38:00.171512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.171469 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf6da91-8216-4983-81e9-70b29346c6e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf6da91-8216-4983-81e9-70b29346c6e9-tls-certs\") pod \"2bf6da91-8216-4983-81e9-70b29346c6e9\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " Oct 30 18:38:00.171689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.171558 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-dshm\") pod \"2bf6da91-8216-4983-81e9-70b29346c6e9\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " Oct 30 18:38:00.171689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.171610 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf6da91-8216-4983-81e9-70b29346c6e9-kube-api-access-d6cwb\" (UniqueName: \"kubernetes.io/projected/2bf6da91-8216-4983-81e9-70b29346c6e9-kube-api-access-d6cwb\") pod \"2bf6da91-8216-4983-81e9-70b29346c6e9\" (UID: \"2bf6da91-8216-4983-81e9-70b29346c6e9\") " Oct 30 18:38:00.171689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.171594 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-model-cache" pod "2bf6da91-8216-4983-81e9-70b29346c6e9" (UID: "2bf6da91-8216-4983-81e9-70b29346c6e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:00.171689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.171661 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-home" pod "2bf6da91-8216-4983-81e9-70b29346c6e9" (UID: "2bf6da91-8216-4983-81e9-70b29346c6e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:00.171889 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.171728 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-tmp-dir" pod "2bf6da91-8216-4983-81e9-70b29346c6e9" (UID: "2bf6da91-8216-4983-81e9-70b29346c6e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:00.172195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.172177 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:00.172296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.172201 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:00.172296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.172240 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:00.173987 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.173966 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf6da91-8216-4983-81e9-70b29346c6e9-kube-api-access-d6cwb" pod "2bf6da91-8216-4983-81e9-70b29346c6e9" (UID: "2bf6da91-8216-4983-81e9-70b29346c6e9"). InnerVolumeSpecName "kube-api-access-d6cwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:00.174081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.173989 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf6da91-8216-4983-81e9-70b29346c6e9-tls-certs" pod "2bf6da91-8216-4983-81e9-70b29346c6e9" (UID: "2bf6da91-8216-4983-81e9-70b29346c6e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:00.174081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.173996 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-dshm" pod "2bf6da91-8216-4983-81e9-70b29346c6e9" (UID: "2bf6da91-8216-4983-81e9-70b29346c6e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:00.273942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.273911 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bf6da91-8216-4983-81e9-70b29346c6e9-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:00.273942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.273937 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bf6da91-8216-4983-81e9-70b29346c6e9-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:00.273942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.273947 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6cwb\" (UniqueName: \"kubernetes.io/projected/2bf6da91-8216-4983-81e9-70b29346c6e9-kube-api-access-d6cwb\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:00.330294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.330234 4862 generic.go:356] "Generic (PLEG): container finished" podID="2bf6da91-8216-4983-81e9-70b29346c6e9" containerID="cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688" exitCode=0 Oct 30 18:38:00.330453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.330313 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" Oct 30 18:38:00.330453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.330314 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" event={"ID":"2bf6da91-8216-4983-81e9-70b29346c6e9","Type":"ContainerDied","Data":"cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688"} Oct 30 18:38:00.330453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.330357 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7" event={"ID":"2bf6da91-8216-4983-81e9-70b29346c6e9","Type":"ContainerDied","Data":"0308f07f3fa80600758327fb7f4535d033b6b7c31eb738b13f6634d26d996673"} Oct 30 18:38:00.330453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.330375 4862 scope.go:122] "RemoveContainer" containerID="cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688" Oct 30 18:38:00.341877 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.341853 4862 scope.go:122] "RemoveContainer" containerID="cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688" Oct 30 18:38:00.342141 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:38:00.342120 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688\": container with ID starting with cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688 not found: ID does not exist" containerID="cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688" Oct 30 18:38:00.342217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.342147 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688"} err="failed to get container status \"cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688\": rpc error: code = NotFound desc = could not find container \"cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688\": container with ID starting with cce9379a25767964a5e2642319ed03fd0755ccfafbce9ce88d3350409a557688 not found: ID does not exist" Oct 30 18:38:00.385422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.385390 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7"] Oct 30 18:38:00.387951 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:00.387929 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4a7508/auth-test-model-4a7508-kserve-5fd7957c9c-b2fx7"] Oct 30 18:38:01.063195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:01.063160 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf6da91-8216-4983-81e9-70b29346c6e9" path="/var/lib/kubelet/pods/2bf6da91-8216-4983-81e9-70b29346c6e9/volumes" Oct 30 18:38:09.117341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:09.117307 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm"] Oct 30 18:38:10.164008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.163975 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq"] Oct 30 18:38:10.164557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164532 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" Oct 30 18:38:10.164557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164545 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164571 4862 cpu_manager.go:414] "containerMap: removing container" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164576 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164590 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2bf6da91-8216-4983-81e9-70b29346c6e9" containerName="main" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164594 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf6da91-8216-4983-81e9-70b29346c6e9" containerName="main" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164603 4862 cpu_manager.go:414] "containerMap: removing container" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164607 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164620 4862 cpu_manager.go:414] "containerMap: removing container" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164625 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164634 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2504490a-8974-45de-a2b9-dc7ec1b2c6c5" containerName="revoke-keys" Oct 30 18:38:10.164669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164638 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2504490a-8974-45de-a2b9-dc7ec1b2c6c5" containerName="revoke-keys" Oct 30 18:38:10.165285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164714 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf6da91-8216-4983-81e9-70b29346c6e9" containerName="main" Oct 30 18:38:10.165285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164722 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="11730e43-29fb-4f70-b591-afed0868b014" containerName="istio-proxy" Oct 30 18:38:10.165285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164730 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d91bf0-629e-4d73-a340-36d38891b3b2" containerName="istio-proxy" Oct 30 18:38:10.165285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164735 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2504490a-8974-45de-a2b9-dc7ec1b2c6c5" containerName="revoke-keys" Oct 30 18:38:10.165285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164744 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d13ba25-0745-413d-8667-3edb42af0102" containerName="istio-proxy" Oct 30 18:38:10.165285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.164750 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="45344089-c81d-4ee0-8615-b6ad76d4027d" containerName="istio-proxy" Oct 30 18:38:10.169436 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.169419 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.197137 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.197116 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:38:10.197253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.197191 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:38:10.206253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.206233 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq"] Oct 30 18:38:10.229323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.229293 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2263df1-wk2nq\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.229554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.229537 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2263df1-wk2nq\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.229636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.229561 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9dk\" (UniqueName: \"kubernetes.io/projected/8b707dba-6650-4aa6-8133-28bbfddc5dae-kube-api-access-bq9dk\") pod \"maas-api-revoke-sub-b2263df1-wk2nq\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.330625 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.330591 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2263df1-wk2nq\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.330625 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.330628 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9dk\" (UniqueName: \"kubernetes.io/projected/8b707dba-6650-4aa6-8133-28bbfddc5dae-kube-api-access-bq9dk\") pod \"maas-api-revoke-sub-b2263df1-wk2nq\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.330879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.330653 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2263df1-wk2nq\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.331216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.331195 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2263df1-wk2nq\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.333150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.333132 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2263df1-wk2nq\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.355919 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.355889 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9dk\" (UniqueName: \"kubernetes.io/projected/8b707dba-6650-4aa6-8133-28bbfddc5dae-kube-api-access-bq9dk\") pod \"maas-api-revoke-sub-b2263df1-wk2nq\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.484748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.484668 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:10.794074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:10.794049 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq"] Oct 30 18:38:10.796412 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:38:10.796382 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b707dba_6650_4aa6_8133_28bbfddc5dae.slice/crio-19b0888e0d40358aebd6301c120474b315b81b18b491228c99495e985a71fe9d WatchSource:0}: Error finding container 19b0888e0d40358aebd6301c120474b315b81b18b491228c99495e985a71fe9d: Status 404 returned error can't find the container with id 19b0888e0d40358aebd6301c120474b315b81b18b491228c99495e985a71fe9d Oct 30 18:38:11.413942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:11.413893 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" event={"ID":"8b707dba-6650-4aa6-8133-28bbfddc5dae","Type":"ContainerStarted","Data":"963da578a15a1a54e3dfa1c265f20b201ffec69281578db9647a8cfcac6c1370"} Oct 30 18:38:11.414387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:11.413985 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" event={"ID":"8b707dba-6650-4aa6-8133-28bbfddc5dae","Type":"ContainerStarted","Data":"19b0888e0d40358aebd6301c120474b315b81b18b491228c99495e985a71fe9d"} Oct 30 18:38:11.461603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:11.461552 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" podStartSLOduration=1.461538121 podStartE2EDuration="1.461538121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:11.4599952 +0000 UTC m=+2390.509040112" watchObservedRunningTime="2026-10-30 18:38:11.461538121 +0000 UTC m=+2390.510583034" Oct 30 18:38:12.422107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:12.422074 4862 generic.go:356] "Generic (PLEG): container finished" podID="8b707dba-6650-4aa6-8133-28bbfddc5dae" containerID="963da578a15a1a54e3dfa1c265f20b201ffec69281578db9647a8cfcac6c1370" exitCode=0 Oct 30 18:38:12.422593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:12.422143 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" event={"ID":"8b707dba-6650-4aa6-8133-28bbfddc5dae","Type":"ContainerDied","Data":"963da578a15a1a54e3dfa1c265f20b201ffec69281578db9647a8cfcac6c1370"} Oct 30 18:38:13.590178 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.590156 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:13.682072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.681979 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-xks-ca\") pod \"8b707dba-6650-4aa6-8133-28bbfddc5dae\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " Oct 30 18:38:13.682239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.682121 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b707dba-6650-4aa6-8133-28bbfddc5dae-kube-api-access-bq9dk\" (UniqueName: \"kubernetes.io/projected/8b707dba-6650-4aa6-8133-28bbfddc5dae-kube-api-access-bq9dk\") pod \"8b707dba-6650-4aa6-8133-28bbfddc5dae\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " Oct 30 18:38:13.682239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.682216 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-openshift-ca\") pod \"8b707dba-6650-4aa6-8133-28bbfddc5dae\" (UID: \"8b707dba-6650-4aa6-8133-28bbfddc5dae\") " Oct 30 18:38:13.682616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.682588 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-openshift-ca" pod "8b707dba-6650-4aa6-8133-28bbfddc5dae" (UID: "8b707dba-6650-4aa6-8133-28bbfddc5dae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:13.683030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.683007 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.684454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.684431 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b707dba-6650-4aa6-8133-28bbfddc5dae-kube-api-access-bq9dk" pod "8b707dba-6650-4aa6-8133-28bbfddc5dae" (UID: "8b707dba-6650-4aa6-8133-28bbfddc5dae"). InnerVolumeSpecName "kube-api-access-bq9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:13.684548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.684497 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-xks-ca" pod "8b707dba-6650-4aa6-8133-28bbfddc5dae" (UID: "8b707dba-6650-4aa6-8133-28bbfddc5dae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:13.783934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.783909 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b707dba-6650-4aa6-8133-28bbfddc5dae-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:13.783934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:13.783933 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9dk\" (UniqueName: \"kubernetes.io/projected/8b707dba-6650-4aa6-8133-28bbfddc5dae-kube-api-access-bq9dk\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:14.436463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:14.436434 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" Oct 30 18:38:14.436648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:14.436459 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq" event={"ID":"8b707dba-6650-4aa6-8133-28bbfddc5dae","Type":"ContainerDied","Data":"19b0888e0d40358aebd6301c120474b315b81b18b491228c99495e985a71fe9d"} Oct 30 18:38:14.436648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:14.436505 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19b0888e0d40358aebd6301c120474b315b81b18b491228c99495e985a71fe9d" Oct 30 18:38:18.097971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:18.097929 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:38:19.871124 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:38:19.871087 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:38:21.534757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:21.534731 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:38:21.536307 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:21.536283 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:38:23.847291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:23.847231 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb"] Oct 30 18:38:23.847706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:23.847463 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" containerID="cri-o://ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68" gracePeriod=30 Oct 30 18:38:24.140208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.140122 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" podUID="629b23f2-dddb-4704-a689-9392ac6942a9" containerName="main" containerID="cri-o://7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842" gracePeriod=45 Oct 30 18:38:24.437513 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.437489 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:38:24.537613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.537580 4862 generic.go:356] "Generic (PLEG): container finished" podID="629b23f2-dddb-4704-a689-9392ac6942a9" containerID="7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842" exitCode=0 Oct 30 18:38:24.537802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.537622 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" Oct 30 18:38:24.537802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.537657 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" event={"ID":"629b23f2-dddb-4704-a689-9392ac6942a9","Type":"ContainerDied","Data":"7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842"} Oct 30 18:38:24.537802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.537699 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm" event={"ID":"629b23f2-dddb-4704-a689-9392ac6942a9","Type":"ContainerDied","Data":"7573d99782be270c3af4ff1300c2753f7f9d6158b7cf5cd555ad4b26e40ac151"} Oct 30 18:38:24.537802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.537716 4862 scope.go:122] "RemoveContainer" containerID="7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842" Oct 30 18:38:24.542249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.542224 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-model-cache\") pod \"629b23f2-dddb-4704-a689-9392ac6942a9\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " Oct 30 18:38:24.542371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.542257 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-tmp-dir\") pod \"629b23f2-dddb-4704-a689-9392ac6942a9\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " Oct 30 18:38:24.542371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.542304 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-home\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-home\") pod \"629b23f2-dddb-4704-a689-9392ac6942a9\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " Oct 30 18:38:24.542371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.542339 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-dshm\") pod \"629b23f2-dddb-4704-a689-9392ac6942a9\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " Oct 30 18:38:24.542480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.542375 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/629b23f2-dddb-4704-a689-9392ac6942a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/629b23f2-dddb-4704-a689-9392ac6942a9-tls-certs\") pod \"629b23f2-dddb-4704-a689-9392ac6942a9\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " Oct 30 18:38:24.542480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.542450 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-model-cache" pod "629b23f2-dddb-4704-a689-9392ac6942a9" (UID: "629b23f2-dddb-4704-a689-9392ac6942a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:24.542569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.542475 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629b23f2-dddb-4704-a689-9392ac6942a9-kube-api-access-2c6kg\" (UniqueName: \"kubernetes.io/projected/629b23f2-dddb-4704-a689-9392ac6942a9-kube-api-access-2c6kg\") pod \"629b23f2-dddb-4704-a689-9392ac6942a9\" (UID: \"629b23f2-dddb-4704-a689-9392ac6942a9\") " Oct 30 18:38:24.542627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.542574 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-home" pod "629b23f2-dddb-4704-a689-9392ac6942a9" (UID: "629b23f2-dddb-4704-a689-9392ac6942a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:24.542767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.542747 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-tmp-dir" pod "629b23f2-dddb-4704-a689-9392ac6942a9" (UID: "629b23f2-dddb-4704-a689-9392ac6942a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:24.543074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.543055 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:24.543134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.543079 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:24.543134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.543088 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:24.545291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.545237 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629b23f2-dddb-4704-a689-9392ac6942a9-tls-certs" pod "629b23f2-dddb-4704-a689-9392ac6942a9" (UID: "629b23f2-dddb-4704-a689-9392ac6942a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:24.545381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.545335 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-dshm" pod "629b23f2-dddb-4704-a689-9392ac6942a9" (UID: "629b23f2-dddb-4704-a689-9392ac6942a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:24.545381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.545368 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629b23f2-dddb-4704-a689-9392ac6942a9-kube-api-access-2c6kg" pod "629b23f2-dddb-4704-a689-9392ac6942a9" (UID: "629b23f2-dddb-4704-a689-9392ac6942a9"). InnerVolumeSpecName "kube-api-access-2c6kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:24.547750 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.547734 4862 scope.go:122] "RemoveContainer" containerID="7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842" Oct 30 18:38:24.548017 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:38:24.547998 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842\": container with ID starting with 7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842 not found: ID does not exist" containerID="7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842" Oct 30 18:38:24.548079 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.548022 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842"} err="failed to get container status \"7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842\": rpc error: code = NotFound desc = could not find container \"7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842\": container with ID starting with 7e1819d5947a7114294e7d8ca0d04be45ccbf10f6e02d59d5323779952ad2842 not found: ID does not exist" Oct 30 18:38:24.644455 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.644428 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/629b23f2-dddb-4704-a689-9392ac6942a9-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:24.644455 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.644453 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/629b23f2-dddb-4704-a689-9392ac6942a9-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:24.644455 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.644463 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c6kg\" (UniqueName: \"kubernetes.io/projected/629b23f2-dddb-4704-a689-9392ac6942a9-kube-api-access-2c6kg\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:24.896355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.896326 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm"] Oct 30 18:38:24.903344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:24.903320 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e95c51/auth-test-model-e95c51-kserve-7ff4994d49-d5rcm"] Oct 30 18:38:25.062073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:25.062046 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629b23f2-dddb-4704-a689-9392ac6942a9" path="/var/lib/kubelet/pods/629b23f2-dddb-4704-a689-9392ac6942a9/volumes" Oct 30 18:38:29.109962 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.109940 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:38:29.207812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.207738 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-socket\") pod \"ca034077-40ef-403c-ad53-f2a0b34524cc\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " Oct 30 18:38:29.207812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.207777 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca034077-40ef-403c-ad53-f2a0b34524cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca034077-40ef-403c-ad53-f2a0b34524cc-istiod-ca-cert\") pod \"ca034077-40ef-403c-ad53-f2a0b34524cc\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " Oct 30 18:38:29.208017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.207855 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-kube-api-access-49hzm\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-kube-api-access-49hzm\") pod \"ca034077-40ef-403c-ad53-f2a0b34524cc\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " Oct 30 18:38:29.208017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.207907 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-credential-socket\") pod \"ca034077-40ef-403c-ad53-f2a0b34524cc\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " Oct 30 18:38:29.208017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.207943 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-token\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-token\") pod \"ca034077-40ef-403c-ad53-f2a0b34524cc\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " Oct 30 18:38:29.208017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.207976 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-envoy\") pod \"ca034077-40ef-403c-ad53-f2a0b34524cc\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " Oct 30 18:38:29.208228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208037 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-podinfo\") pod \"ca034077-40ef-403c-ad53-f2a0b34524cc\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " Oct 30 18:38:29.208228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208039 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-socket" pod "ca034077-40ef-403c-ad53-f2a0b34524cc" (UID: "ca034077-40ef-403c-ad53-f2a0b34524cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:29.208228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208106 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-data\") pod \"ca034077-40ef-403c-ad53-f2a0b34524cc\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " Oct 30 18:38:29.208228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208205 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-certs\") pod \"ca034077-40ef-403c-ad53-f2a0b34524cc\" (UID: \"ca034077-40ef-403c-ad53-f2a0b34524cc\") " Oct 30 18:38:29.208471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208294 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-credential-socket" pod "ca034077-40ef-403c-ad53-f2a0b34524cc" (UID: "ca034077-40ef-403c-ad53-f2a0b34524cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:29.208471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208304 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca034077-40ef-403c-ad53-f2a0b34524cc-istiod-ca-cert" pod "ca034077-40ef-403c-ad53-f2a0b34524cc" (UID: "ca034077-40ef-403c-ad53-f2a0b34524cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:29.209226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208598 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-data" pod "ca034077-40ef-403c-ad53-f2a0b34524cc" (UID: "ca034077-40ef-403c-ad53-f2a0b34524cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:29.209226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208860 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.209226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208881 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.209226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208895 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.209226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.208909 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca034077-40ef-403c-ad53-f2a0b34524cc-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.209226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.209154 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-certs" pod "ca034077-40ef-403c-ad53-f2a0b34524cc" (UID: "ca034077-40ef-403c-ad53-f2a0b34524cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:29.210877 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.210852 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-kube-api-access-49hzm" pod "ca034077-40ef-403c-ad53-f2a0b34524cc" (UID: "ca034077-40ef-403c-ad53-f2a0b34524cc"). InnerVolumeSpecName "kube-api-access-49hzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:29.210989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.210900 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-envoy" pod "ca034077-40ef-403c-ad53-f2a0b34524cc" (UID: "ca034077-40ef-403c-ad53-f2a0b34524cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:38:29.211087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.211067 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-podinfo" pod "ca034077-40ef-403c-ad53-f2a0b34524cc" (UID: "ca034077-40ef-403c-ad53-f2a0b34524cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:38:29.211251 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.211235 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-token" pod "ca034077-40ef-403c-ad53-f2a0b34524cc" (UID: "ca034077-40ef-403c-ad53-f2a0b34524cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:29.310088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.310054 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49hzm\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-kube-api-access-49hzm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.310088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.310084 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.310088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.310094 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.310368 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.310103 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca034077-40ef-403c-ad53-f2a0b34524cc-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.310368 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.310111 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca034077-40ef-403c-ad53-f2a0b34524cc-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.571821 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.571793 4862 generic.go:356] "Generic (PLEG): container finished" podID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerID="ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68" exitCode=0 Oct 30 18:38:29.572009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.571837 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" Oct 30 18:38:29.572009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.571875 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" event={"ID":"ca034077-40ef-403c-ad53-f2a0b34524cc","Type":"ContainerDied","Data":"ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68"} Oct 30 18:38:29.572009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.571939 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb" event={"ID":"ca034077-40ef-403c-ad53-f2a0b34524cc","Type":"ContainerDied","Data":"10361fcb4387e216706f4f02d0e633e81dd4d00d7f7c9bd78e5d6577e6508dcc"} Oct 30 18:38:29.572009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.571968 4862 scope.go:122] "RemoveContainer" containerID="ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68" Oct 30 18:38:29.582723 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.582709 4862 scope.go:122] "RemoveContainer" containerID="ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68" Oct 30 18:38:29.583007 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:38:29.582981 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68\": container with ID starting with ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68 not found: ID does not exist" containerID="ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68" Oct 30 18:38:29.583115 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.583012 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68"} err="failed to get container status \"ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68\": rpc error: code = NotFound desc = could not find container \"ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68\": container with ID starting with ec1a0de322bb0b1f99a0ac6b29c4f7b6d01b5f841e3c8122d038f5fea5d3bc68 not found: ID does not exist" Oct 30 18:38:29.626023 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.626001 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb"] Oct 30 18:38:29.638025 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:29.637997 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-wm7rb"] Oct 30 18:38:31.062476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:38:31.062445 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" path="/var/lib/kubelet/pods/ca034077-40ef-403c-ad53-f2a0b34524cc/volumes" Oct 30 18:38:49.869497 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:38:49.869456 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:39:16.568526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:16.568492 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:39:19.872064 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:39:19.872026 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:39:31.162093 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162058 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd"] Oct 30 18:39:31.162595 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162580 4862 cpu_manager.go:414] "containerMap: removing container" podUID="629b23f2-dddb-4704-a689-9392ac6942a9" containerName="main" Oct 30 18:39:31.162595 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162596 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629b23f2-dddb-4704-a689-9392ac6942a9" containerName="main" Oct 30 18:39:31.162694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162605 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8b707dba-6650-4aa6-8133-28bbfddc5dae" containerName="revoke-keys" Oct 30 18:39:31.162694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162610 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b707dba-6650-4aa6-8133-28bbfddc5dae" containerName="revoke-keys" Oct 30 18:39:31.162694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162658 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" Oct 30 18:39:31.162694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162665 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" Oct 30 18:39:31.162843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162739 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="629b23f2-dddb-4704-a689-9392ac6942a9" containerName="main" Oct 30 18:39:31.162843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162748 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b707dba-6650-4aa6-8133-28bbfddc5dae" containerName="revoke-keys" Oct 30 18:39:31.162843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.162757 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca034077-40ef-403c-ad53-f2a0b34524cc" containerName="istio-proxy" Oct 30 18:39:31.167679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.167661 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:31.184413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.184387 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd"] Oct 30 18:39:31.195737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.193459 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-38e0db\"" Oct 30 18:39:31.291013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.290973 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a2fa1-7a97-488b-8420-e9074e752891-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd\" (UID: \"2c3a2fa1-7a97-488b-8420-e9074e752891\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:31.291206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.291023 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4nwg\" (UniqueName: \"kubernetes.io/projected/2c3a2fa1-7a97-488b-8420-e9074e752891-kube-api-access-g4nwg\") pod \"payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd\" (UID: \"2c3a2fa1-7a97-488b-8420-e9074e752891\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:31.392159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.392119 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a2fa1-7a97-488b-8420-e9074e752891-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd\" (UID: \"2c3a2fa1-7a97-488b-8420-e9074e752891\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:31.392449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.392426 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4nwg\" (UniqueName: \"kubernetes.io/projected/2c3a2fa1-7a97-488b-8420-e9074e752891-kube-api-access-g4nwg\") pod \"payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd\" (UID: \"2c3a2fa1-7a97-488b-8420-e9074e752891\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:31.393111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.393083 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a2fa1-7a97-488b-8420-e9074e752891-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd\" (UID: \"2c3a2fa1-7a97-488b-8420-e9074e752891\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:31.412533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.412476 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4nwg\" (UniqueName: \"kubernetes.io/projected/2c3a2fa1-7a97-488b-8420-e9074e752891-kube-api-access-g4nwg\") pod \"payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd\" (UID: \"2c3a2fa1-7a97-488b-8420-e9074e752891\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:31.482085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.482063 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:31.786713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:31.786687 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd"] Oct 30 18:39:31.788014 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:39:31.787989 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3a2fa1_7a97_488b_8420_e9074e752891.slice/crio-de8c4c11e8ab81d523615b8cb78bb593b3973ff0be072a42eb66fa8294bee2a8 WatchSource:0}: Error finding container de8c4c11e8ab81d523615b8cb78bb593b3973ff0be072a42eb66fa8294bee2a8: Status 404 returned error can't find the container with id de8c4c11e8ab81d523615b8cb78bb593b3973ff0be072a42eb66fa8294bee2a8 Oct 30 18:39:32.003409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:32.003369 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" event={"ID":"2c3a2fa1-7a97-488b-8420-e9074e752891","Type":"ContainerStarted","Data":"385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8"} Oct 30 18:39:32.003565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:32.003435 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:32.003565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:32.003456 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" event={"ID":"2c3a2fa1-7a97-488b-8420-e9074e752891","Type":"ContainerStarted","Data":"de8c4c11e8ab81d523615b8cb78bb593b3973ff0be072a42eb66fa8294bee2a8"} Oct 30 18:39:32.042241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:32.042161 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" podStartSLOduration=1.042148698 podStartE2EDuration="1.042148698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:32.041722406 +0000 UTC m=+2471.090767321" watchObservedRunningTime="2026-10-30 18:39:32.042148698 +0000 UTC m=+2471.091193665" Oct 30 18:39:43.013563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:43.013531 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 18:39:49.901615 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:39:49.901576 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:39:55.499045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.499011 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs"] Oct 30 18:39:55.502900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.502878 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:39:55.531358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.531330 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-713dc8-dockercfg-v5rsw\"" Oct 30 18:39:55.531540 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.531403 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-713dc8\"" Oct 30 18:39:55.534448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.534421 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs"] Oct 30 18:39:55.620657 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.620615 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8345433c-4916-4e21-af2a-9a5214ed504d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs\" (UID: \"8345433c-4916-4e21-af2a-9a5214ed504d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:39:55.620974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.620955 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6f68\" (UniqueName: \"kubernetes.io/projected/8345433c-4916-4e21-af2a-9a5214ed504d-kube-api-access-h6f68\") pod \"payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs\" (UID: \"8345433c-4916-4e21-af2a-9a5214ed504d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:39:55.721985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.721951 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8345433c-4916-4e21-af2a-9a5214ed504d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs\" (UID: \"8345433c-4916-4e21-af2a-9a5214ed504d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:39:55.722149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.722076 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6f68\" (UniqueName: \"kubernetes.io/projected/8345433c-4916-4e21-af2a-9a5214ed504d-kube-api-access-h6f68\") pod \"payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs\" (UID: \"8345433c-4916-4e21-af2a-9a5214ed504d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:39:55.722747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.722727 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8345433c-4916-4e21-af2a-9a5214ed504d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs\" (UID: \"8345433c-4916-4e21-af2a-9a5214ed504d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:39:55.741902 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.741873 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6f68\" (UniqueName: \"kubernetes.io/projected/8345433c-4916-4e21-af2a-9a5214ed504d-kube-api-access-h6f68\") pod \"payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs\" (UID: \"8345433c-4916-4e21-af2a-9a5214ed504d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:39:55.817448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:55.817426 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:39:56.119097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:56.119067 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs"] Oct 30 18:39:56.120201 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:39:56.120176 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8345433c_4916_4e21_af2a_9a5214ed504d.slice/crio-494f2b1ff6be32b4c45424b6fb4151920ab34dfe61aa6f42e46e8e54ea6e7f97 WatchSource:0}: Error finding container 494f2b1ff6be32b4c45424b6fb4151920ab34dfe61aa6f42e46e8e54ea6e7f97: Status 404 returned error can't find the container with id 494f2b1ff6be32b4c45424b6fb4151920ab34dfe61aa6f42e46e8e54ea6e7f97 Oct 30 18:39:56.148498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:56.148461 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" event={"ID":"8345433c-4916-4e21-af2a-9a5214ed504d","Type":"ContainerStarted","Data":"494f2b1ff6be32b4c45424b6fb4151920ab34dfe61aa6f42e46e8e54ea6e7f97"} Oct 30 18:39:57.072434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.072407 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d"] Oct 30 18:39:57.076296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.076279 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:39:57.093531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.093506 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d"] Oct 30 18:39:57.102406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.102385 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-861bb3\"" Oct 30 18:39:57.119576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.119553 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q"] Oct 30 18:39:57.123748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.123727 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:39:57.139063 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.138988 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41efe939-9a37-451e-b6d6-3a47877fbe13-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d\" (UID: \"41efe939-9a37-451e-b6d6-3a47877fbe13\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:39:57.139420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.139395 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8pjg\" (UniqueName: \"kubernetes.io/projected/b86db503-6cf0-4171-8020-a08639061418-kube-api-access-b8pjg\") pod \"payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q\" (UID: \"b86db503-6cf0-4171-8020-a08639061418\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:39:57.139648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.139622 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b2br\" (UniqueName: \"kubernetes.io/projected/41efe939-9a37-451e-b6d6-3a47877fbe13-kube-api-access-4b2br\") pod \"payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d\" (UID: \"41efe939-9a37-451e-b6d6-3a47877fbe13\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:39:57.140010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.139982 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86db503-6cf0-4171-8020-a08639061418-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q\" (UID: \"b86db503-6cf0-4171-8020-a08639061418\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:39:57.141908 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.141889 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-861bb3-dockercfg-w95sg\"" Oct 30 18:39:57.149283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.147838 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q"] Oct 30 18:39:57.157640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.157614 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" event={"ID":"8345433c-4916-4e21-af2a-9a5214ed504d","Type":"ContainerStarted","Data":"3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6"} Oct 30 18:39:57.158302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.158247 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:39:57.205720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.205650 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" podStartSLOduration=2.205629128 podStartE2EDuration="2.205629128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:57.203662611 +0000 UTC m=+2496.252707551" watchObservedRunningTime="2026-10-30 18:39:57.205629128 +0000 UTC m=+2496.254674041" Oct 30 18:39:57.249285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.242162 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86db503-6cf0-4171-8020-a08639061418-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q\" (UID: \"b86db503-6cf0-4171-8020-a08639061418\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:39:57.249285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.242308 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41efe939-9a37-451e-b6d6-3a47877fbe13-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d\" (UID: \"41efe939-9a37-451e-b6d6-3a47877fbe13\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:39:57.249285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.242381 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8pjg\" (UniqueName: \"kubernetes.io/projected/b86db503-6cf0-4171-8020-a08639061418-kube-api-access-b8pjg\") pod \"payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q\" (UID: \"b86db503-6cf0-4171-8020-a08639061418\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:39:57.249285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.242502 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b2br\" (UniqueName: \"kubernetes.io/projected/41efe939-9a37-451e-b6d6-3a47877fbe13-kube-api-access-4b2br\") pod \"payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d\" (UID: \"41efe939-9a37-451e-b6d6-3a47877fbe13\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:39:57.272284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.257839 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41efe939-9a37-451e-b6d6-3a47877fbe13-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d\" (UID: \"41efe939-9a37-451e-b6d6-3a47877fbe13\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:39:57.287281 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.281482 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86db503-6cf0-4171-8020-a08639061418-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q\" (UID: \"b86db503-6cf0-4171-8020-a08639061418\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:39:57.288082 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.288051 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8pjg\" (UniqueName: \"kubernetes.io/projected/b86db503-6cf0-4171-8020-a08639061418-kube-api-access-b8pjg\") pod \"payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q\" (UID: \"b86db503-6cf0-4171-8020-a08639061418\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:39:57.293009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.292925 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b2br\" (UniqueName: \"kubernetes.io/projected/41efe939-9a37-451e-b6d6-3a47877fbe13-kube-api-access-4b2br\") pod \"payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d\" (UID: \"41efe939-9a37-451e-b6d6-3a47877fbe13\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:39:57.389329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.389238 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:39:57.438853 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.438814 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:39:57.701097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.701069 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d"] Oct 30 18:39:57.702398 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:39:57.702373 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41efe939_9a37_451e_b6d6_3a47877fbe13.slice/crio-879b08a54e77a7f9b394cb559411482d692017d4bd34b04d208fb19619311168 WatchSource:0}: Error finding container 879b08a54e77a7f9b394cb559411482d692017d4bd34b04d208fb19619311168: Status 404 returned error can't find the container with id 879b08a54e77a7f9b394cb559411482d692017d4bd34b04d208fb19619311168 Oct 30 18:39:57.777371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:57.777346 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q"] Oct 30 18:39:57.778447 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:39:57.778421 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86db503_6cf0_4171_8020_a08639061418.slice/crio-ec5a647cbd989a5ef82c041103d640372852cad6672b696675fd49713902588a WatchSource:0}: Error finding container ec5a647cbd989a5ef82c041103d640372852cad6672b696675fd49713902588a: Status 404 returned error can't find the container with id ec5a647cbd989a5ef82c041103d640372852cad6672b696675fd49713902588a Oct 30 18:39:58.165552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:58.165512 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" event={"ID":"41efe939-9a37-451e-b6d6-3a47877fbe13","Type":"ContainerStarted","Data":"b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa"} Oct 30 18:39:58.166029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:58.165582 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:39:58.166029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:58.165603 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" event={"ID":"41efe939-9a37-451e-b6d6-3a47877fbe13","Type":"ContainerStarted","Data":"879b08a54e77a7f9b394cb559411482d692017d4bd34b04d208fb19619311168"} Oct 30 18:39:58.167663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:58.167633 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" event={"ID":"b86db503-6cf0-4171-8020-a08639061418","Type":"ContainerStarted","Data":"5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511"} Oct 30 18:39:58.167833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:58.167683 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" event={"ID":"b86db503-6cf0-4171-8020-a08639061418","Type":"ContainerStarted","Data":"ec5a647cbd989a5ef82c041103d640372852cad6672b696675fd49713902588a"} Oct 30 18:39:58.167833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:58.167821 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:39:58.206276 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:58.206217 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" podStartSLOduration=1.206204896 podStartE2EDuration="1.206204896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:58.202832989 +0000 UTC m=+2497.251877902" watchObservedRunningTime="2026-10-30 18:39:58.206204896 +0000 UTC m=+2497.255249809" Oct 30 18:39:58.242168 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:39:58.242128 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" podStartSLOduration=1.242115155 podStartE2EDuration="1.242115155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:58.240019576 +0000 UTC m=+2497.289064488" watchObservedRunningTime="2026-10-30 18:39:58.242115155 +0000 UTC m=+2497.291160068" Oct 30 18:40:03.693284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:03.693232 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:40:05.039981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.039650 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm"] Oct 30 18:40:05.063491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.063459 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.085373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.085223 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-8b8e54-openshift-default-dockercfg-q4lxn\"" Oct 30 18:40:05.087612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.087586 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm"] Oct 30 18:40:05.167175 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.167134 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6gn5\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-kube-api-access-r6gn5\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.167525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.167499 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebbe758-a515-41cd-a91d-05d4647b74e5-istiod-ca-cert\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.167802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.167779 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-envoy\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.168034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.167996 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-data\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.168277 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.168242 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-certs\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.169144 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.169122 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-credential-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.169498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.169468 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-token\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.169720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.169698 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.170017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.169988 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-podinfo\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.277291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.277126 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebbe758-a515-41cd-a91d-05d4647b74e5-istiod-ca-cert\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.277479 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.277300 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-envoy\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.277479 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.277381 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-data\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.277479 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.277442 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-certs\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.277661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.277514 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-credential-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.277661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.277542 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-token\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.277661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.277609 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.277661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.277635 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-podinfo\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.277865 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.277709 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6gn5\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-kube-api-access-r6gn5\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.286288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.278808 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebbe758-a515-41cd-a91d-05d4647b74e5-istiod-ca-cert\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.286288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.281867 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-envoy\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.286288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.282379 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.286288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.283607 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-data\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.286288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.283840 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-certs\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.286288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.285398 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-credential-socket\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.291727 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.290222 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-podinfo\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.313347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.305546 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6gn5\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-kube-api-access-r6gn5\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.313347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.309952 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-token\") pod \"e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.394304 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.394180 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:05.718960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.718936 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm"] Oct 30 18:40:05.719635 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:40:05.719613 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebbe758_a515_41cd_a91d_05d4647b74e5.slice/crio-93eb4fc01441a71bc7a9bab11878364125b96e8b31cfd11cadf7a147d2403ef4 WatchSource:0}: Error finding container 93eb4fc01441a71bc7a9bab11878364125b96e8b31cfd11cadf7a147d2403ef4: Status 404 returned error can't find the container with id 93eb4fc01441a71bc7a9bab11878364125b96e8b31cfd11cadf7a147d2403ef4 Oct 30 18:40:05.722554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.722522 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:05.722643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.722587 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:05.722643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:05.722621 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:40:06.235050 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:06.235014 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" event={"ID":"bebbe758-a515-41cd-a91d-05d4647b74e5","Type":"ContainerStarted","Data":"15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc"} Oct 30 18:40:06.235566 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:06.235063 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" event={"ID":"bebbe758-a515-41cd-a91d-05d4647b74e5","Type":"ContainerStarted","Data":"93eb4fc01441a71bc7a9bab11878364125b96e8b31cfd11cadf7a147d2403ef4"} Oct 30 18:40:06.278606 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:06.278551 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" podStartSLOduration=1.278538069 podStartE2EDuration="1.278538069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:06.276245185 +0000 UTC m=+2505.325290101" watchObservedRunningTime="2026-10-30 18:40:06.278538069 +0000 UTC m=+2505.327582982" Oct 30 18:40:06.395284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:06.394989 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:06.459976 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:06.459645 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:07.240833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:07.240807 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:07.241828 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:07.241807 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:40:09.175606 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:09.175577 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 18:40:09.175971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:09.175647 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 18:40:09.175971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:09.175811 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 18:40:10.646915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.646882 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn"] Oct 30 18:40:10.651192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.651176 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:10.671583 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.671557 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn"] Oct 30 18:40:10.679296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.679274 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-8b8e54\"" Oct 30 18:40:10.679433 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.679302 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-8b8e54-dockercfg-spq4w\"" Oct 30 18:40:10.794693 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.772168 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p84cc\" (UniqueName: \"kubernetes.io/projected/a6df5a42-1fc7-4159-b61a-ab0c969e5985-kube-api-access-p84cc\") pod \"payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn\" (UID: \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:10.794693 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.772383 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6df5a42-1fc7-4159-b61a-ab0c969e5985-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn\" (UID: \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:10.874031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.873994 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6df5a42-1fc7-4159-b61a-ab0c969e5985-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn\" (UID: \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:10.874206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.874107 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p84cc\" (UniqueName: \"kubernetes.io/projected/a6df5a42-1fc7-4159-b61a-ab0c969e5985-kube-api-access-p84cc\") pod \"payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn\" (UID: \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:10.874721 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.874700 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6df5a42-1fc7-4159-b61a-ab0c969e5985-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn\" (UID: \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:10.893242 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.893199 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p84cc\" (UniqueName: \"kubernetes.io/projected/a6df5a42-1fc7-4159-b61a-ab0c969e5985-kube-api-access-p84cc\") pod \"payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn\" (UID: \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:10.964944 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:10.964882 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:11.271726 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:11.271704 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn"] Oct 30 18:40:11.273826 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:40:11.273798 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6df5a42_1fc7_4159_b61a_ab0c969e5985.slice/crio-609c6a61f9ab462e7ee9507169bb3340c3ce72ac33fdefe208ce78912a594060 WatchSource:0}: Error finding container 609c6a61f9ab462e7ee9507169bb3340c3ce72ac33fdefe208ce78912a594060: Status 404 returned error can't find the container with id 609c6a61f9ab462e7ee9507169bb3340c3ce72ac33fdefe208ce78912a594060 Oct 30 18:40:12.273617 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:12.273575 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" event={"ID":"a6df5a42-1fc7-4159-b61a-ab0c969e5985","Type":"ContainerStarted","Data":"1b764ef0fde7e2805147f265c8af7576856f2cc19b870f6a90ff7c3a33c16ffd"} Oct 30 18:40:12.274087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:12.273640 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:12.274087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:12.273660 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" event={"ID":"a6df5a42-1fc7-4159-b61a-ab0c969e5985","Type":"ContainerStarted","Data":"609c6a61f9ab462e7ee9507169bb3340c3ce72ac33fdefe208ce78912a594060"} Oct 30 18:40:12.313691 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:12.313646 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" podStartSLOduration=2.313630606 podStartE2EDuration="2.313630606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:12.312288305 +0000 UTC m=+2511.361333221" watchObservedRunningTime="2026-10-30 18:40:12.313630606 +0000 UTC m=+2511.362675521" Oct 30 18:40:19.870056 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:40:19.870015 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:40:23.282349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:23.282324 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 18:40:39.503813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:39.503782 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml"] Oct 30 18:40:39.504331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:39.504031 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" containerID="cri-o://4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2" gracePeriod=30 Oct 30 18:40:39.516524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:39.516495 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7"] Oct 30 18:40:39.516752 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:39.516730 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" containerID="cri-o://de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5" gracePeriod=30 Oct 30 18:40:44.181759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.181729 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5"] Oct 30 18:40:44.186518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.186501 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:44.205922 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.205902 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5"] Oct 30 18:40:44.215251 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.215213 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6babca-dockercfg-29v87\"" Oct 30 18:40:44.259659 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.259626 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6babca\"" Oct 30 18:40:44.324355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.324317 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33431f9-a4e7-4691-9fbb-1226b6483fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5\" (UID: \"b33431f9-a4e7-4691-9fbb-1226b6483fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:44.325659 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.325635 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5vx\" (UniqueName: \"kubernetes.io/projected/b33431f9-a4e7-4691-9fbb-1226b6483fab-kube-api-access-dh5vx\") pod \"payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5\" (UID: \"b33431f9-a4e7-4691-9fbb-1226b6483fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:44.426912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.426875 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5vx\" (UniqueName: \"kubernetes.io/projected/b33431f9-a4e7-4691-9fbb-1226b6483fab-kube-api-access-dh5vx\") pod \"payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5\" (UID: \"b33431f9-a4e7-4691-9fbb-1226b6483fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:44.427159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.427139 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33431f9-a4e7-4691-9fbb-1226b6483fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5\" (UID: \"b33431f9-a4e7-4691-9fbb-1226b6483fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:44.428216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.428071 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33431f9-a4e7-4691-9fbb-1226b6483fab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5\" (UID: \"b33431f9-a4e7-4691-9fbb-1226b6483fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:44.447017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.446947 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5vx\" (UniqueName: \"kubernetes.io/projected/b33431f9-a4e7-4691-9fbb-1226b6483fab-kube-api-access-dh5vx\") pod \"payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5\" (UID: \"b33431f9-a4e7-4691-9fbb-1226b6483fab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:44.501244 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.501218 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:44.793044 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.793020 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:40:44.824230 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.824205 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:40:44.826602 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.826571 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5"] Oct 30 18:40:44.945117 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945090 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de62f70d-94b4-4121-9095-efbda0423381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de62f70d-94b4-4121-9095-efbda0423381-istio-podinfo\") pod \"de62f70d-94b4-4121-9095-efbda0423381\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " Oct 30 18:40:44.945256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945163 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de62f70d-94b4-4121-9095-efbda0423381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de62f70d-94b4-4121-9095-efbda0423381-istiod-ca-cert\") pod \"de62f70d-94b4-4121-9095-efbda0423381\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " Oct 30 18:40:44.945256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945211 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-certs\") pod \"f51f6875-e361-4f37-ba52-70fa15ac77ea\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " Oct 30 18:40:44.945256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945239 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-envoy\") pod \"de62f70d-94b4-4121-9095-efbda0423381\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " Oct 30 18:40:44.945456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945289 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-socket\") pod \"f51f6875-e361-4f37-ba52-70fa15ac77ea\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " Oct 30 18:40:44.945456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945323 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-data\") pod \"f51f6875-e361-4f37-ba52-70fa15ac77ea\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " Oct 30 18:40:44.945456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945354 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-certs\") pod \"de62f70d-94b4-4121-9095-efbda0423381\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " Oct 30 18:40:44.945456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945392 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-kube-api-access-j8hjn\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-kube-api-access-j8hjn\") pod \"de62f70d-94b4-4121-9095-efbda0423381\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " Oct 30 18:40:44.945456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945439 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-socket\") pod \"de62f70d-94b4-4121-9095-efbda0423381\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " Oct 30 18:40:44.945715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945513 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-certs" pod "f51f6875-e361-4f37-ba52-70fa15ac77ea" (UID: "f51f6875-e361-4f37-ba52-70fa15ac77ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.945715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945543 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-kube-api-access-8cb95\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-kube-api-access-8cb95\") pod \"f51f6875-e361-4f37-ba52-70fa15ac77ea\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " Oct 30 18:40:44.945715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945586 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-socket" pod "f51f6875-e361-4f37-ba52-70fa15ac77ea" (UID: "f51f6875-e361-4f37-ba52-70fa15ac77ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.945715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945613 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51f6875-e361-4f37-ba52-70fa15ac77ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f6875-e361-4f37-ba52-70fa15ac77ea-istiod-ca-cert\") pod \"f51f6875-e361-4f37-ba52-70fa15ac77ea\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " Oct 30 18:40:44.945715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945650 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-istio-token\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-istio-token\") pod \"de62f70d-94b4-4121-9095-efbda0423381\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " Oct 30 18:40:44.945715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945647 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-data" pod "f51f6875-e361-4f37-ba52-70fa15ac77ea" (UID: "f51f6875-e361-4f37-ba52-70fa15ac77ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.946028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945762 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-credential-socket\") pod \"de62f70d-94b4-4121-9095-efbda0423381\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " Oct 30 18:40:44.946028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945797 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-envoy\") pod \"f51f6875-e361-4f37-ba52-70fa15ac77ea\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " Oct 30 18:40:44.946028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945861 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-credential-socket\") pod \"f51f6875-e361-4f37-ba52-70fa15ac77ea\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " Oct 30 18:40:44.946028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945906 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-token\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-token\") pod \"f51f6875-e361-4f37-ba52-70fa15ac77ea\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " Oct 30 18:40:44.946028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945932 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de62f70d-94b4-4121-9095-efbda0423381-istiod-ca-cert" pod "de62f70d-94b4-4121-9095-efbda0423381" (UID: "de62f70d-94b4-4121-9095-efbda0423381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:44.946028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945963 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-socket" pod "de62f70d-94b4-4121-9095-efbda0423381" (UID: "de62f70d-94b4-4121-9095-efbda0423381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.946028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.945965 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-podinfo\") pod \"f51f6875-e361-4f37-ba52-70fa15ac77ea\" (UID: \"f51f6875-e361-4f37-ba52-70fa15ac77ea\") " Oct 30 18:40:44.946827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.946090 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-data\") pod \"de62f70d-94b4-4121-9095-efbda0423381\" (UID: \"de62f70d-94b4-4121-9095-efbda0423381\") " Oct 30 18:40:44.946827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.946127 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-credential-socket" pod "de62f70d-94b4-4121-9095-efbda0423381" (UID: "de62f70d-94b4-4121-9095-efbda0423381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.946827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.946140 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-certs" pod "de62f70d-94b4-4121-9095-efbda0423381" (UID: "de62f70d-94b4-4121-9095-efbda0423381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.946827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.946419 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-credential-socket" pod "f51f6875-e361-4f37-ba52-70fa15ac77ea" (UID: "f51f6875-e361-4f37-ba52-70fa15ac77ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.946827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.946698 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51f6875-e361-4f37-ba52-70fa15ac77ea-istiod-ca-cert" pod "f51f6875-e361-4f37-ba52-70fa15ac77ea" (UID: "f51f6875-e361-4f37-ba52-70fa15ac77ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:44.947101 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.946903 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-data" pod "de62f70d-94b4-4121-9095-efbda0423381" (UID: "de62f70d-94b4-4121-9095-efbda0423381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.947101 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.946928 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:44.947101 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.946951 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:44.947305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.946966 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de62f70d-94b4-4121-9095-efbda0423381-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:44.947305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.947167 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:44.947305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.947179 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:44.947305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.947195 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:44.947305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.947209 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:44.947305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.947237 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:44.947305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.947250 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f6875-e361-4f37-ba52-70fa15ac77ea-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:44.948635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.948548 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-kube-api-access-j8hjn" pod "de62f70d-94b4-4121-9095-efbda0423381" (UID: "de62f70d-94b4-4121-9095-efbda0423381"). InnerVolumeSpecName "kube-api-access-j8hjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:44.948732 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.948651 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de62f70d-94b4-4121-9095-efbda0423381-istio-podinfo" pod "de62f70d-94b4-4121-9095-efbda0423381" (UID: "de62f70d-94b4-4121-9095-efbda0423381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:44.949288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.949205 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-envoy" pod "de62f70d-94b4-4121-9095-efbda0423381" (UID: "de62f70d-94b4-4121-9095-efbda0423381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.949288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.949238 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-token" pod "f51f6875-e361-4f37-ba52-70fa15ac77ea" (UID: "f51f6875-e361-4f37-ba52-70fa15ac77ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:44.949705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.949675 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-kube-api-access-8cb95" pod "f51f6875-e361-4f37-ba52-70fa15ac77ea" (UID: "f51f6875-e361-4f37-ba52-70fa15ac77ea"). InnerVolumeSpecName "kube-api-access-8cb95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:44.949804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.949786 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-podinfo" pod "f51f6875-e361-4f37-ba52-70fa15ac77ea" (UID: "f51f6875-e361-4f37-ba52-70fa15ac77ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:40:44.950000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.949982 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-envoy" pod "f51f6875-e361-4f37-ba52-70fa15ac77ea" (UID: "f51f6875-e361-4f37-ba52-70fa15ac77ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:40:44.950085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:44.950068 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-istio-token" pod "de62f70d-94b4-4121-9095-efbda0423381" (UID: "de62f70d-94b4-4121-9095-efbda0423381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:45.048761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.048728 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:45.048761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.048760 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:45.048761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.048769 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f6875-e361-4f37-ba52-70fa15ac77ea-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:45.048996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.048778 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:45.048996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.048786 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de62f70d-94b4-4121-9095-efbda0423381-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:45.048996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.048794 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de62f70d-94b4-4121-9095-efbda0423381-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:45.048996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.048803 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8hjn\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-kube-api-access-j8hjn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:45.048996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.048814 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cb95\" (UniqueName: \"kubernetes.io/projected/f51f6875-e361-4f37-ba52-70fa15ac77ea-kube-api-access-8cb95\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:45.048996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.048822 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de62f70d-94b4-4121-9095-efbda0423381-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:40:45.492145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.489075 4862 generic.go:356] "Generic (PLEG): container finished" podID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerID="de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5" exitCode=0 Oct 30 18:40:45.492145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.489302 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" event={"ID":"f51f6875-e361-4f37-ba52-70fa15ac77ea","Type":"ContainerDied","Data":"de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5"} Oct 30 18:40:45.492145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.489379 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" event={"ID":"f51f6875-e361-4f37-ba52-70fa15ac77ea","Type":"ContainerDied","Data":"81735286fb861fe545acfa5160f406062341771f2607d8d88a8f48f93781832d"} Oct 30 18:40:45.492145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.489413 4862 scope.go:122] "RemoveContainer" containerID="de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5" Oct 30 18:40:45.492145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.489623 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7" Oct 30 18:40:45.493343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.493307 4862 generic.go:356] "Generic (PLEG): container finished" podID="de62f70d-94b4-4121-9095-efbda0423381" containerID="4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2" exitCode=0 Oct 30 18:40:45.493489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.493443 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" event={"ID":"de62f70d-94b4-4121-9095-efbda0423381","Type":"ContainerDied","Data":"4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2"} Oct 30 18:40:45.493489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.493480 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" event={"ID":"de62f70d-94b4-4121-9095-efbda0423381","Type":"ContainerDied","Data":"9eb01ea42112e92c9109e9f9d99116001f89d131054c1d83bdec8d9c8a75d58d"} Oct 30 18:40:45.493717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.493698 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml" Oct 30 18:40:45.495535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.495509 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" event={"ID":"b33431f9-a4e7-4691-9fbb-1226b6483fab","Type":"ContainerStarted","Data":"9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59"} Oct 30 18:40:45.495651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.495559 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" event={"ID":"b33431f9-a4e7-4691-9fbb-1226b6483fab","Type":"ContainerStarted","Data":"f13bb482217b5f1a069c07aa02efa98f74442cd6ceb228d2f3945b55cc903942"} Oct 30 18:40:45.495651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.495591 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:45.502024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.502006 4862 scope.go:122] "RemoveContainer" containerID="de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5" Oct 30 18:40:45.502320 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:40:45.502296 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5\": container with ID starting with de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5 not found: ID does not exist" containerID="de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5" Oct 30 18:40:45.502408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.502331 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5"} err="failed to get container status \"de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5\": rpc error: code = NotFound desc = could not find container \"de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5\": container with ID starting with de0cda354e884c1c47627fe16b1eb0f3f1bf9c2ea3cc2587a097a820fd1a63f5 not found: ID does not exist" Oct 30 18:40:45.502408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.502355 4862 scope.go:122] "RemoveContainer" containerID="4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2" Oct 30 18:40:45.511927 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.511908 4862 scope.go:122] "RemoveContainer" containerID="4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2" Oct 30 18:40:45.512177 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:40:45.512156 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2\": container with ID starting with 4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2 not found: ID does not exist" containerID="4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2" Oct 30 18:40:45.512250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.512185 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2"} err="failed to get container status \"4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2\": rpc error: code = NotFound desc = could not find container \"4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2\": container with ID starting with 4eff2188adcba971c9812d5fe1025508f39ab10de4ce1fcae88b44a03b0017e2 not found: ID does not exist" Oct 30 18:40:45.554824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.554795 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7"] Oct 30 18:40:45.561517 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.561493 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-mkqg7"] Oct 30 18:40:45.592838 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.592816 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml"] Oct 30 18:40:45.596885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.596865 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-de7e3a-openshift-default-557f55b6c7-tzxml"] Oct 30 18:40:45.633059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:45.633020 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" podStartSLOduration=1.633007847 podStartE2EDuration="1.633007847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:45.631702096 +0000 UTC m=+2544.680747009" watchObservedRunningTime="2026-10-30 18:40:45.633007847 +0000 UTC m=+2544.682052759" Oct 30 18:40:47.062630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:47.062601 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de62f70d-94b4-4121-9095-efbda0423381" path="/var/lib/kubelet/pods/de62f70d-94b4-4121-9095-efbda0423381/volumes" Oct 30 18:40:47.063096 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:47.063052 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" path="/var/lib/kubelet/pods/f51f6875-e361-4f37-ba52-70fa15ac77ea/volumes" Oct 30 18:40:49.869513 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:40:49.869481 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:40:50.597736 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:50.597694 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:40:51.129986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.129948 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd"] Oct 30 18:40:51.130730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.130709 4862 cpu_manager.go:414] "containerMap: removing container" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" Oct 30 18:40:51.130838 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.130732 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" Oct 30 18:40:51.130838 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.130810 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" Oct 30 18:40:51.130838 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.130819 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" Oct 30 18:40:51.130989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.130957 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51f6875-e361-4f37-ba52-70fa15ac77ea" containerName="istio-proxy" Oct 30 18:40:51.130989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.130974 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="de62f70d-94b4-4121-9095-efbda0423381" containerName="istio-proxy" Oct 30 18:40:51.134676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.134655 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.152372 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.152350 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-8b8e54-serving-cert\"" Oct 30 18:40:51.171663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.171601 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd"] Oct 30 18:40:51.224323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.224292 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-metrics-certs\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.224475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.224417 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.224568 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.224552 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbqm\" (UniqueName: \"kubernetes.io/projected/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-kube-api-access-hgbqm\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.224630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.224617 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.326028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.325997 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-metrics-certs\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.326220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.326059 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.326220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.326093 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbqm\" (UniqueName: \"kubernetes.io/projected/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-kube-api-access-hgbqm\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.326220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.326135 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.326873 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.326793 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.328750 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.328730 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-maas-api-tls\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.329009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.328987 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-metrics-certs\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.345502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.345482 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbqm\" (UniqueName: \"kubernetes.io/projected/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-kube-api-access-hgbqm\") pod \"maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.449325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.449224 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:51.788313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.784993 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd"] Oct 30 18:40:51.788313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:51.785523 4862 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:40:52.551753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:52.551713 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" event={"ID":"f8dc3f79-83b1-4e35-92f4-6a13a39834e3","Type":"ContainerStarted","Data":"ce415efe65901e698fee487b49e39f644004399e6967c6a72205044786af653b"} Oct 30 18:40:52.552491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:52.551783 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" event={"ID":"f8dc3f79-83b1-4e35-92f4-6a13a39834e3","Type":"ContainerStarted","Data":"2b7027cfabdf8d73254247bd7c0d72823b381cd6a7f6266c8b45af7665c86fd6"} Oct 30 18:40:52.552491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:52.551814 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:40:52.619287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:52.615406 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" podStartSLOduration=1.225924365 podStartE2EDuration="1.615386003s" totalImagesPullingTime="389.461638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:52.614664633 +0000 UTC m=+2551.663709546" watchObservedRunningTime="2026-10-30 18:40:52.615386003 +0000 UTC m=+2551.664430917" Oct 30 18:40:56.508165 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:56.508136 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 18:40:58.569749 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:40:58.569720 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 18:41:10.448703 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.448619 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j"] Oct 30 18:41:10.454003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.453979 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.470866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.470844 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j"] Oct 30 18:41:10.480826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.480159 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"agc-cred-d71fbdf35f65\"" Oct 30 18:41:10.480826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.480174 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"payload-processing-external-model-e2e--d1be02e2-dockercfg-cr6xg\"" Oct 30 18:41:10.480826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.480299 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"routing-overlay\"" Oct 30 18:41:10.480826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.480385 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"kube-root-ca.crt\"" Oct 30 18:41:10.480826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.480798 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-38e0db\"" Oct 30 18:41:10.480826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.480827 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"openshift-service-ca.crt\"" Oct 30 18:41:10.592058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.592022 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqdv\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-kube-api-access-zwqdv\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.592279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.592147 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.592279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.592219 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.592279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.592240 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.693886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.693850 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.693886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.693886 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.694126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.693969 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqdv\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-kube-api-access-zwqdv\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.694126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.694044 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.694694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.694621 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.694694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.694652 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.696656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.696631 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.714222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.714164 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqdv\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-kube-api-access-zwqdv\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:10.768828 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:10.768801 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:11.077825 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:11.077800 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j"] Oct 30 18:41:11.081185 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:41:11.081153 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91feb246_58aa_448b_9d12_9b97d7589271.slice/crio-20d6e6d78336dc0493d8099ab3b3d16b2b955017b65b6a2b1f13b875819c0d17 WatchSource:0}: Error finding container 20d6e6d78336dc0493d8099ab3b3d16b2b955017b65b6a2b1f13b875819c0d17: Status 404 returned error can't find the container with id 20d6e6d78336dc0493d8099ab3b3d16b2b955017b65b6a2b1f13b875819c0d17 Oct 30 18:41:11.678863 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:11.678823 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" event={"ID":"91feb246-58aa-448b-9d12-9b97d7589271","Type":"ContainerStarted","Data":"76cd77cfc76d6c9e5613ffbba71ec21320cf3db2ac99d35a2bd238d791c416b9"} Oct 30 18:41:11.679315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:11.678886 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" event={"ID":"91feb246-58aa-448b-9d12-9b97d7589271","Type":"ContainerStarted","Data":"20d6e6d78336dc0493d8099ab3b3d16b2b955017b65b6a2b1f13b875819c0d17"} Oct 30 18:41:11.679315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:11.678915 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:11.720229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:11.720182 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" podStartSLOduration=1.7201711149999999 podStartE2EDuration="1.720171115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:11.719172987 +0000 UTC m=+2570.768217900" watchObservedRunningTime="2026-10-30 18:41:11.720171115 +0000 UTC m=+2570.769216028" Oct 30 18:41:19.870399 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:41:19.870361 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:41:22.687998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:22.687966 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:41:24.489876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:24.489843 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn"] Oct 30 18:41:24.490358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:24.490051 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" containerID="cri-o://f4e1655fa8eec133bfef33fdd10a62fe7d00c083a139bb6bc0e965cd0573c827" gracePeriod=30 Oct 30 18:41:25.362291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:25.350004 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:41:27.216097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.216062 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn"] Oct 30 18:41:27.222428 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.222405 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.240058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.240027 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn"] Oct 30 18:41:27.248303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.246936 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-39c99d-openshift-default-dockercfg-6b8ss\"" Oct 30 18:41:27.356284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.347087 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-envoy\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.356284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.347152 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-credential-socket\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.356284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.347240 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-podinfo\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.356284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.347399 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-certs\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.356284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.347430 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/757179a3-982f-402c-8b8a-00acf2cc8f56-istiod-ca-cert\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.356284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.347464 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-socket\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.356284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.347513 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw5d9\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-kube-api-access-bw5d9\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.356284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.347625 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-data\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.356284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.347657 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-token\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.462913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.462873 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-socket\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.464209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.464182 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw5d9\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-kube-api-access-bw5d9\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.464857 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.464835 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-data\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.465019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.465002 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-token\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.465181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.465162 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-envoy\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.467945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.467590 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-data\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.467945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.463746 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-socket\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.484305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.472338 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-credential-socket\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.484305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.472419 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-podinfo\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.484305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.472684 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-certs\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.484305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.472711 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/757179a3-982f-402c-8b8a-00acf2cc8f56-istiod-ca-cert\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.484305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.473484 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/757179a3-982f-402c-8b8a-00acf2cc8f56-istiod-ca-cert\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.484305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.473625 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-certs\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.484305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.473906 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-credential-socket\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.494715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.493462 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-token\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.516651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.516585 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-envoy\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.529313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.517841 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-podinfo\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.529782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.529757 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw5d9\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-kube-api-access-bw5d9\") pod \"e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.540908 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.540871 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:27.955293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.953929 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn"] Oct 30 18:41:27.962625 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.962587 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:27.962754 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.962659 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:27.962754 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:27.962690 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:28.813255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:28.813215 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" event={"ID":"757179a3-982f-402c-8b8a-00acf2cc8f56","Type":"ContainerStarted","Data":"e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894"} Oct 30 18:41:28.813766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:28.813311 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" event={"ID":"757179a3-982f-402c-8b8a-00acf2cc8f56","Type":"ContainerStarted","Data":"b15cdd6c67ebe9f180ba8591bbf9c7d999320f8b7c9f4666dcad57c313433efc"} Oct 30 18:41:29.553287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:29.549465 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:29.579288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:29.570555 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:29.626473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:29.626419 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" podStartSLOduration=2.626400585 podStartE2EDuration="2.626400585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:28.890535398 +0000 UTC m=+2587.939580338" watchObservedRunningTime="2026-10-30 18:41:29.626400585 +0000 UTC m=+2588.675445499" Oct 30 18:41:29.880493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:29.880459 4862 generic.go:356] "Generic (PLEG): container finished" podID="bfa04282-d984-44f3-a442-655532f052af" containerID="f4e1655fa8eec133bfef33fdd10a62fe7d00c083a139bb6bc0e965cd0573c827" exitCode=0 Oct 30 18:41:29.882915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:29.881027 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" event={"ID":"bfa04282-d984-44f3-a442-655532f052af","Type":"ContainerDied","Data":"f4e1655fa8eec133bfef33fdd10a62fe7d00c083a139bb6bc0e965cd0573c827"} Oct 30 18:41:29.883242 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:29.883228 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:29.884493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:29.884477 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:41:29.983775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:29.983746 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:41:30.026129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.026096 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-envoy\") pod \"bfa04282-d984-44f3-a442-655532f052af\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " Oct 30 18:41:30.026321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.026186 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-socket\") pod \"bfa04282-d984-44f3-a442-655532f052af\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " Oct 30 18:41:30.026321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.026231 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-credential-socket\") pod \"bfa04282-d984-44f3-a442-655532f052af\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " Oct 30 18:41:30.026321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.026292 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-kube-api-access-q4bc4\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-kube-api-access-q4bc4\") pod \"bfa04282-d984-44f3-a442-655532f052af\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " Oct 30 18:41:30.026548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.026354 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa04282-d984-44f3-a442-655532f052af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa04282-d984-44f3-a442-655532f052af-istiod-ca-cert\") pod \"bfa04282-d984-44f3-a442-655532f052af\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " Oct 30 18:41:30.026548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.026403 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfa04282-d984-44f3-a442-655532f052af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa04282-d984-44f3-a442-655532f052af-istio-podinfo\") pod \"bfa04282-d984-44f3-a442-655532f052af\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " Oct 30 18:41:30.026548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.026436 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-data\") pod \"bfa04282-d984-44f3-a442-655532f052af\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " Oct 30 18:41:30.026548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.026509 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-certs\") pod \"bfa04282-d984-44f3-a442-655532f052af\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " Oct 30 18:41:30.027782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.027253 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-credential-socket" pod "bfa04282-d984-44f3-a442-655532f052af" (UID: "bfa04282-d984-44f3-a442-655532f052af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.027782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.027444 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-socket" pod "bfa04282-d984-44f3-a442-655532f052af" (UID: "bfa04282-d984-44f3-a442-655532f052af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.027782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.027536 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa04282-d984-44f3-a442-655532f052af-istiod-ca-cert" pod "bfa04282-d984-44f3-a442-655532f052af" (UID: "bfa04282-d984-44f3-a442-655532f052af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:30.028140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.027881 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-data" pod "bfa04282-d984-44f3-a442-655532f052af" (UID: "bfa04282-d984-44f3-a442-655532f052af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.028140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.028054 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-certs" pod "bfa04282-d984-44f3-a442-655532f052af" (UID: "bfa04282-d984-44f3-a442-655532f052af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.028243 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.028213 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-istio-token\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-istio-token\") pod \"bfa04282-d984-44f3-a442-655532f052af\" (UID: \"bfa04282-d984-44f3-a442-655532f052af\") " Oct 30 18:41:30.065042 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.065009 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.066085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.066063 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.066218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.066205 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa04282-d984-44f3-a442-655532f052af-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.066331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.066318 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.066437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.066426 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.067319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.067295 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-kube-api-access-q4bc4" pod "bfa04282-d984-44f3-a442-655532f052af" (UID: "bfa04282-d984-44f3-a442-655532f052af"). InnerVolumeSpecName "kube-api-access-q4bc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:30.067478 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.067407 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfa04282-d984-44f3-a442-655532f052af-istio-podinfo" pod "bfa04282-d984-44f3-a442-655532f052af" (UID: "bfa04282-d984-44f3-a442-655532f052af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:41:30.069495 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.069462 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-envoy" pod "bfa04282-d984-44f3-a442-655532f052af" (UID: "bfa04282-d984-44f3-a442-655532f052af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:30.072227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.072203 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-istio-token" pod "bfa04282-d984-44f3-a442-655532f052af" (UID: "bfa04282-d984-44f3-a442-655532f052af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:30.168416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.168320 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.168416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.168352 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa04282-d984-44f3-a442-655532f052af-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.168416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.168364 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4bc4\" (UniqueName: \"kubernetes.io/projected/bfa04282-d984-44f3-a442-655532f052af-kube-api-access-q4bc4\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.168416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.168377 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa04282-d984-44f3-a442-655532f052af-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:41:30.889236 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.889208 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" Oct 30 18:41:30.889759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.889246 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn" event={"ID":"bfa04282-d984-44f3-a442-655532f052af","Type":"ContainerDied","Data":"d89bffed60190a9ab95494b5eb00715d7feeba6d2461e4ea434ff8272e3f7f81"} Oct 30 18:41:30.889759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.889328 4862 scope.go:122] "RemoveContainer" containerID="f4e1655fa8eec133bfef33fdd10a62fe7d00c083a139bb6bc0e965cd0573c827" Oct 30 18:41:30.976123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.976088 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn"] Oct 30 18:41:30.979126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:30.979096 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-n4tnn"] Oct 30 18:41:31.064236 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:31.064206 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa04282-d984-44f3-a442-655532f052af" path="/var/lib/kubelet/pods/bfa04282-d984-44f3-a442-655532f052af/volumes" Oct 30 18:41:33.223975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.223906 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2"] Oct 30 18:41:33.224496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.224482 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" Oct 30 18:41:33.224549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.224497 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" Oct 30 18:41:33.224598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.224589 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa04282-d984-44f3-a442-655532f052af" containerName="istio-proxy" Oct 30 18:41:33.229460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.229434 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:33.247328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.247301 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2"] Oct 30 18:41:33.255024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.255001 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-39c99d\"" Oct 30 18:41:33.308906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.308871 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xp82\" (UniqueName: \"kubernetes.io/projected/0e4bee85-2a79-4470-a777-dc2ebb574684-kube-api-access-7xp82\") pod \"payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2\" (UID: \"0e4bee85-2a79-4470-a777-dc2ebb574684\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:33.309072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.308996 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4bee85-2a79-4470-a777-dc2ebb574684-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2\" (UID: \"0e4bee85-2a79-4470-a777-dc2ebb574684\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:33.412098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.410507 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xp82\" (UniqueName: \"kubernetes.io/projected/0e4bee85-2a79-4470-a777-dc2ebb574684-kube-api-access-7xp82\") pod \"payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2\" (UID: \"0e4bee85-2a79-4470-a777-dc2ebb574684\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:33.412098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.410607 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4bee85-2a79-4470-a777-dc2ebb574684-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2\" (UID: \"0e4bee85-2a79-4470-a777-dc2ebb574684\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:33.412098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.411666 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4bee85-2a79-4470-a777-dc2ebb574684-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2\" (UID: \"0e4bee85-2a79-4470-a777-dc2ebb574684\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:33.465918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.456963 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xp82\" (UniqueName: \"kubernetes.io/projected/0e4bee85-2a79-4470-a777-dc2ebb574684-kube-api-access-7xp82\") pod \"payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2\" (UID: \"0e4bee85-2a79-4470-a777-dc2ebb574684\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:33.543501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.543418 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:33.858347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.858318 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2"] Oct 30 18:41:33.860414 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:41:33.860379 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4bee85_2a79_4470_a777_dc2ebb574684.slice/crio-c213e8b4094bbd4ed206a3de949a043025a53e4625fab7f62df5d899a1597d87 WatchSource:0}: Error finding container c213e8b4094bbd4ed206a3de949a043025a53e4625fab7f62df5d899a1597d87: Status 404 returned error can't find the container with id c213e8b4094bbd4ed206a3de949a043025a53e4625fab7f62df5d899a1597d87 Oct 30 18:41:33.912924 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:33.912893 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" event={"ID":"0e4bee85-2a79-4470-a777-dc2ebb574684","Type":"ContainerStarted","Data":"c213e8b4094bbd4ed206a3de949a043025a53e4625fab7f62df5d899a1597d87"} Oct 30 18:41:34.921806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:34.921775 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" event={"ID":"0e4bee85-2a79-4470-a777-dc2ebb574684","Type":"ContainerStarted","Data":"09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58"} Oct 30 18:41:34.922213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:34.921976 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:34.963238 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:34.963191 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" podStartSLOduration=1.9631779059999999 podStartE2EDuration="1.963177906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:34.961057354 +0000 UTC m=+2594.010102267" watchObservedRunningTime="2026-10-30 18:41:34.963177906 +0000 UTC m=+2594.012222818" Oct 30 18:41:36.935926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:36.935847 4862 generic.go:356] "Generic (PLEG): container finished" podID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerID="15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc" exitCode=0 Oct 30 18:41:36.936438 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:36.935922 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" event={"ID":"bebbe758-a515-41cd-a91d-05d4647b74e5","Type":"ContainerDied","Data":"15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc"} Oct 30 18:41:36.936438 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:36.936312 4862 scope.go:122] "RemoveContainer" containerID="15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc" Oct 30 18:41:36.937961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:36.937935 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:36.938029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:36.938006 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:36.938064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:36.938037 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:37.246127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:37.243137 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:41:37.959852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:37.959816 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" event={"ID":"bebbe758-a515-41cd-a91d-05d4647b74e5","Type":"ContainerStarted","Data":"5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c"} Oct 30 18:41:38.133413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.133378 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q"] Oct 30 18:41:38.185937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.185842 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q"] Oct 30 18:41:38.186139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.185957 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.202282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.201204 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6018b4-openshift-default-dockercfg-l97jn\"" Oct 30 18:41:38.286620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.286527 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a741c85-d6fc-4bac-82c1-58884544c340-istio-podinfo\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.286620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.286591 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-envoy\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.286854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.286696 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-socket\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.286854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.286761 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-data\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.286928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.286871 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-credential-socket\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.286968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.286935 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-istio-token\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.286968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.286952 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a741c85-d6fc-4bac-82c1-58884544c340-istiod-ca-cert\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.287073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.286968 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-certs\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.287132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.287062 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-kube-api-access-tcqh8\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.390713 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-data\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.390836 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-credential-socket\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.390928 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-istio-token\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.390952 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a741c85-d6fc-4bac-82c1-58884544c340-istiod-ca-cert\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.390977 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-certs\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.391009 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-kube-api-access-tcqh8\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.391093 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a741c85-d6fc-4bac-82c1-58884544c340-istio-podinfo\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.391143 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-envoy\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.391189 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-socket\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.391692 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-socket\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.392309 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-data\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.394346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.392558 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-credential-socket\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.395139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.394855 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-certs\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.429299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.411328 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:41:38.429299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.411485 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:41:38.429299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.411533 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 18:41:38.429299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.414006 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a741c85-d6fc-4bac-82c1-58884544c340-istiod-ca-cert\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.429299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.423953 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-envoy\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.477881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.465157 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-istio-token\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.477881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.465577 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a741c85-d6fc-4bac-82c1-58884544c340-istio-podinfo\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.477881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.475091 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-kube-api-access-tcqh8\") pod \"e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.509545 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.509512 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:38.940360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.940302 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q"] Oct 30 18:41:38.956290 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:41:38.953808 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a741c85_d6fc_4bac_82c1_58884544c340.slice/crio-dee3e216c1eedec19cae027ef94f54a6d00de4e50dad1bbee823da789001631b WatchSource:0}: Error finding container dee3e216c1eedec19cae027ef94f54a6d00de4e50dad1bbee823da789001631b: Status 404 returned error can't find the container with id dee3e216c1eedec19cae027ef94f54a6d00de4e50dad1bbee823da789001631b Oct 30 18:41:38.964901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.964859 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:38.965371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.964956 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:38.965371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.965000 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:41:38.969107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:38.969072 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" event={"ID":"6a741c85-d6fc-4bac-82c1-58884544c340","Type":"ContainerStarted","Data":"dee3e216c1eedec19cae027ef94f54a6d00de4e50dad1bbee823da789001631b"} Oct 30 18:41:39.394465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:39.394423 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:41:39.394654 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:39.394510 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 18:41:39.980810 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:39.980776 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" event={"ID":"6a741c85-d6fc-4bac-82c1-58884544c340","Type":"ContainerStarted","Data":"366f11bbdc5f36eebbeabbe5400169b0b9c58a78da49e93a27eac5726b9c54ff"} Oct 30 18:41:40.394804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:40.394766 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:41:40.394979 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:40.394833 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 18:41:40.509779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:40.509750 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:40.515805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:40.515781 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:40.562067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:40.562007 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" podStartSLOduration=2.5619901990000002 podStartE2EDuration="2.561990199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:40.053559241 +0000 UTC m=+2599.102604155" watchObservedRunningTime="2026-10-30 18:41:40.561990199 +0000 UTC m=+2599.611035114" Oct 30 18:41:40.987750 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:40.987721 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:40.988752 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:40.988729 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:41:41.395034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:41.394995 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:41:41.395223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:41.395055 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 18:41:42.395126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:42.395089 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:41:42.395636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:42.395142 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 30 18:41:43.699798 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:43.699767 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:41:43.700345 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:43.700249 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:41:43.701282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:43.701241 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 18:41:44.660283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.630311 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2"] Oct 30 18:41:44.660283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.643717 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:44.667047 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.662105 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6018b4\"" Oct 30 18:41:44.672956 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.672914 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2"] Oct 30 18:41:44.675313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.674565 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6018b4-dockercfg-r77lp\"" Oct 30 18:41:44.713287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.712810 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2\" (UID: \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:44.713287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.713179 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwd7\" (UniqueName: \"kubernetes.io/projected/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-kube-api-access-cnwd7\") pod \"payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2\" (UID: \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:44.814383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.814356 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwd7\" (UniqueName: \"kubernetes.io/projected/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-kube-api-access-cnwd7\") pod \"payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2\" (UID: \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:44.814591 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.814439 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2\" (UID: \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:44.815200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.815180 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2\" (UID: \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:44.833510 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.833485 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwd7\" (UniqueName: \"kubernetes.io/projected/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-kube-api-access-cnwd7\") pod \"payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2\" (UID: \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:44.996415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:44.996345 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:45.329330 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:45.329303 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2"] Oct 30 18:41:45.329950 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:41:45.329921 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579a0de2_def5_43cf_8fa4_7fb58cea0f4e.slice/crio-1fe0c18495ed7bd4eeba1770091afa3293d298a937c043eff1998b6c11cea19a WatchSource:0}: Error finding container 1fe0c18495ed7bd4eeba1770091afa3293d298a937c043eff1998b6c11cea19a: Status 404 returned error can't find the container with id 1fe0c18495ed7bd4eeba1770091afa3293d298a937c043eff1998b6c11cea19a Oct 30 18:41:45.931550 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:45.931524 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 18:41:46.024636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:46.024598 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" event={"ID":"579a0de2-def5-43cf-8fa4-7fb58cea0f4e","Type":"ContainerStarted","Data":"adfd1dff8020ee79466e508138c34462fe0c0d864ebe10a705d5615ca7816184"} Oct 30 18:41:46.024822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:46.024664 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" event={"ID":"579a0de2-def5-43cf-8fa4-7fb58cea0f4e","Type":"ContainerStarted","Data":"1fe0c18495ed7bd4eeba1770091afa3293d298a937c043eff1998b6c11cea19a"} Oct 30 18:41:46.024822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:46.024693 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:46.064765 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:46.064720 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" podStartSLOduration=2.0647067 podStartE2EDuration="2.0647067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:46.063137668 +0000 UTC m=+2605.112182581" watchObservedRunningTime="2026-10-30 18:41:46.0647067 +0000 UTC m=+2605.113751613" Oct 30 18:41:49.870055 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:41:49.870008 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:41:55.833513 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.833479 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn"] Oct 30 18:41:55.837943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.837927 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:55.857928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.857902 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn"] Oct 30 18:41:55.865421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.865398 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"routing-overlay\"" Oct 30 18:41:55.865585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.865561 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"agc-cred-695bf65e6f2e\"" Oct 30 18:41:55.865661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.865581 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"payload-processing-external-model-e2e--13a66b56-dockercfg-kjqxg\"" Oct 30 18:41:55.865661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.865413 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8b8e54\"" Oct 30 18:41:55.865770 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.865660 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"openshift-service-ca.crt\"" Oct 30 18:41:55.867470 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.867437 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"kube-root-ca.crt\"" Oct 30 18:41:55.988502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.988466 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4pc\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-kube-api-access-rz4pc\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:55.996182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:55.996150 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.000433 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.000401 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.000704 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.000683 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.102615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.102540 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4pc\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-kube-api-access-rz4pc\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.102775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.102638 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.102775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.102690 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.102892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.102792 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.103562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.103498 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.103687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.103577 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.105346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.105324 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.121703 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.121680 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4pc\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-kube-api-access-rz4pc\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.151574 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.151552 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:56.462825 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:56.461252 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn"] Oct 30 18:41:56.470932 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:41:56.470898 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd083e4_1455_4a3e_856f_0b7947d931e4.slice/crio-cafce3fd5e55348b9ff1f7c799928e34be6078969212bea74568958dbcc3ce61 WatchSource:0}: Error finding container cafce3fd5e55348b9ff1f7c799928e34be6078969212bea74568958dbcc3ce61: Status 404 returned error can't find the container with id cafce3fd5e55348b9ff1f7c799928e34be6078969212bea74568958dbcc3ce61 Oct 30 18:41:57.046561 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:57.046522 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 18:41:57.097588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:57.097550 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" event={"ID":"fdd083e4-1455-4a3e-856f-0b7947d931e4","Type":"ContainerStarted","Data":"29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952"} Oct 30 18:41:57.097588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:57.097593 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" event={"ID":"fdd083e4-1455-4a3e-856f-0b7947d931e4","Type":"ContainerStarted","Data":"cafce3fd5e55348b9ff1f7c799928e34be6078969212bea74568958dbcc3ce61"} Oct 30 18:41:57.097837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:57.097683 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:41:57.137893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:41:57.137846 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" podStartSLOduration=2.13783256 podStartE2EDuration="2.13783256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:57.136387801 +0000 UTC m=+2616.185432714" watchObservedRunningTime="2026-10-30 18:41:57.13783256 +0000 UTC m=+2616.186877473" Oct 30 18:42:00.029910 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:00.029876 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs"] Oct 30 18:42:00.041776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:00.041750 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6737f14b-zbnxs"] Oct 30 18:42:01.062652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:01.062623 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2504490a-8974-45de-a2b9-dc7ec1b2c6c5" path="/var/lib/kubelet/pods/2504490a-8974-45de-a2b9-dc7ec1b2c6c5/volumes" Oct 30 18:42:08.107463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:08.107438 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:42:09.713609 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:09.713578 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq"] Oct 30 18:42:09.714007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:09.713769 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" containerID="cri-o://d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30" gracePeriod=30 Oct 30 18:42:14.981809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:14.981786 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:42:15.138715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.138678 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-credential-socket\") pod \"30b0d537-406a-479c-8deb-6f2afdfdce5a\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " Oct 30 18:42:15.138880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.138738 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b0d537-406a-479c-8deb-6f2afdfdce5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b0d537-406a-479c-8deb-6f2afdfdce5a-istiod-ca-cert\") pod \"30b0d537-406a-479c-8deb-6f2afdfdce5a\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " Oct 30 18:42:15.138880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.138776 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-token\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-token\") pod \"30b0d537-406a-479c-8deb-6f2afdfdce5a\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " Oct 30 18:42:15.138880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.138795 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-data\") pod \"30b0d537-406a-479c-8deb-6f2afdfdce5a\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " Oct 30 18:42:15.139026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.138875 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-kube-api-access-jf7q2\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-kube-api-access-jf7q2\") pod \"30b0d537-406a-479c-8deb-6f2afdfdce5a\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " Oct 30 18:42:15.139026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.138925 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-credential-socket" pod "30b0d537-406a-479c-8deb-6f2afdfdce5a" (UID: "30b0d537-406a-479c-8deb-6f2afdfdce5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:15.139026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.138944 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-certs\") pod \"30b0d537-406a-479c-8deb-6f2afdfdce5a\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " Oct 30 18:42:15.139026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.138980 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-envoy\") pod \"30b0d537-406a-479c-8deb-6f2afdfdce5a\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " Oct 30 18:42:15.139026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.139008 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-socket\") pod \"30b0d537-406a-479c-8deb-6f2afdfdce5a\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " Oct 30 18:42:15.139317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.139034 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-podinfo\") pod \"30b0d537-406a-479c-8deb-6f2afdfdce5a\" (UID: \"30b0d537-406a-479c-8deb-6f2afdfdce5a\") " Oct 30 18:42:15.139317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.139121 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-data" pod "30b0d537-406a-479c-8deb-6f2afdfdce5a" (UID: "30b0d537-406a-479c-8deb-6f2afdfdce5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:15.139437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.139314 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-certs" pod "30b0d537-406a-479c-8deb-6f2afdfdce5a" (UID: "30b0d537-406a-479c-8deb-6f2afdfdce5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:15.139437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.139404 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b0d537-406a-479c-8deb-6f2afdfdce5a-istiod-ca-cert" pod "30b0d537-406a-479c-8deb-6f2afdfdce5a" (UID: "30b0d537-406a-479c-8deb-6f2afdfdce5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:15.139437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.139406 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-socket" pod "30b0d537-406a-479c-8deb-6f2afdfdce5a" (UID: "30b0d537-406a-479c-8deb-6f2afdfdce5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:15.140048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.140027 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.140166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.140051 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.140166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.140064 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.140166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.140077 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30b0d537-406a-479c-8deb-6f2afdfdce5a-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.140166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.140089 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.141790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.141769 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-kube-api-access-jf7q2" pod "30b0d537-406a-479c-8deb-6f2afdfdce5a" (UID: "30b0d537-406a-479c-8deb-6f2afdfdce5a"). InnerVolumeSpecName "kube-api-access-jf7q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:15.142637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.142616 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-podinfo" pod "30b0d537-406a-479c-8deb-6f2afdfdce5a" (UID: "30b0d537-406a-479c-8deb-6f2afdfdce5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:42:15.142637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.142627 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-token" pod "30b0d537-406a-479c-8deb-6f2afdfdce5a" (UID: "30b0d537-406a-479c-8deb-6f2afdfdce5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:15.142776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.142628 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-envoy" pod "30b0d537-406a-479c-8deb-6f2afdfdce5a" (UID: "30b0d537-406a-479c-8deb-6f2afdfdce5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:15.218175 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.218151 4862 generic.go:356] "Generic (PLEG): container finished" podID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerID="d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30" exitCode=0 Oct 30 18:42:15.218322 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.218200 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" Oct 30 18:42:15.218322 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.218236 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" event={"ID":"30b0d537-406a-479c-8deb-6f2afdfdce5a","Type":"ContainerDied","Data":"d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30"} Oct 30 18:42:15.218322 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.218294 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq" event={"ID":"30b0d537-406a-479c-8deb-6f2afdfdce5a","Type":"ContainerDied","Data":"4d58b0e4ec84427f54fde920fb2c2a5bc235f0ad1749990075b9e14938382a4e"} Oct 30 18:42:15.218322 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.218322 4862 scope.go:122] "RemoveContainer" containerID="d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30" Oct 30 18:42:15.229535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.229518 4862 scope.go:122] "RemoveContainer" containerID="d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30" Oct 30 18:42:15.229827 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:42:15.229810 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30\": container with ID starting with d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30 not found: ID does not exist" containerID="d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30" Oct 30 18:42:15.229894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.229836 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30"} err="failed to get container status \"d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30\": rpc error: code = NotFound desc = could not find container \"d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30\": container with ID starting with d58d03c6138c5cf65c49a60b5378c4073921b16560a011522426d9bfc2dc6d30 not found: ID does not exist" Oct 30 18:42:15.240806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.240788 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.240896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.240810 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7q2\" (UniqueName: \"kubernetes.io/projected/30b0d537-406a-479c-8deb-6f2afdfdce5a-kube-api-access-jf7q2\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.240896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.240822 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.240896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.240831 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30b0d537-406a-479c-8deb-6f2afdfdce5a-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.270348 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.270323 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq"] Oct 30 18:42:15.298767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.294698 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-jp4hq"] Oct 30 18:42:15.645812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:15.645776 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:42:17.063735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:17.063706 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" path="/var/lib/kubelet/pods/30b0d537-406a-479c-8deb-6f2afdfdce5a/volumes" Oct 30 18:42:19.871402 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:42:19.871360 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:42:43.802536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.802504 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7"] Oct 30 18:42:43.803124 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.803109 4862 cpu_manager.go:414] "containerMap: removing container" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" Oct 30 18:42:43.803168 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.803126 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" Oct 30 18:42:43.803244 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.803234 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b0d537-406a-479c-8deb-6f2afdfdce5a" containerName="istio-proxy" Oct 30 18:42:43.835892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.835863 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7"] Oct 30 18:42:43.836045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.835985 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:43.850926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.850903 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6babca\"/\"kube-root-ca.crt\"" Oct 30 18:42:43.851097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.850988 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6babca\"/\"openshift-service-ca.crt\"" Oct 30 18:42:43.851097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.850995 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6babca\"/\"agc-cred-86a75f599ff5\"" Oct 30 18:42:43.863391 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.863344 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6babca\"/\"payload-processing-external-model-e2e--e1df0121-dockercfg-xfmxt\"" Oct 30 18:42:43.863530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.863395 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6babca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6babca\"" Oct 30 18:42:43.863530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.863463 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6babca\"/\"routing-overlay\"" Oct 30 18:42:43.934848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.934377 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtm5\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-kube-api-access-hjtm5\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:43.934848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.934464 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:43.934848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.934555 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:43.934848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:43.934612 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.036356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.036322 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.036569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.036490 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtm5\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-kube-api-access-hjtm5\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.036569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.036529 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.036714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.036604 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.037234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.037205 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.037380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.037361 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.039183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.039160 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.056862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.056802 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtm5\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-kube-api-access-hjtm5\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769b964n7\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.164453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.164426 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:44.473322 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:44.473291 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7"] Oct 30 18:42:44.474581 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:42:44.474545 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b0150b_579c_4ea4_abf8_506418c68165.slice/crio-bb70ef547b480796e708d79bb42fb95351c95a49a14962b6d0816e63c799190b WatchSource:0}: Error finding container bb70ef547b480796e708d79bb42fb95351c95a49a14962b6d0816e63c799190b: Status 404 returned error can't find the container with id bb70ef547b480796e708d79bb42fb95351c95a49a14962b6d0816e63c799190b Oct 30 18:42:45.348132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.346033 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": read tcp 10.133.0.2:35122->10.133.0.119:15021: read: connection reset by peer" start-of-body= Oct 30 18:42:45.348132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.346095 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": read tcp 10.133.0.2:35122->10.133.0.119:15021: read: connection reset by peer" Oct 30 18:42:45.470457 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.470418 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" event={"ID":"13b0150b-579c-4ea4-abf8-506418c68165","Type":"ContainerStarted","Data":"ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5"} Oct 30 18:42:45.470672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.470482 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" event={"ID":"13b0150b-579c-4ea4-abf8-506418c68165","Type":"ContainerStarted","Data":"bb70ef547b480796e708d79bb42fb95351c95a49a14962b6d0816e63c799190b"} Oct 30 18:42:45.470672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.470641 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:42:45.472757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.472731 4862 generic.go:356] "Generic (PLEG): container finished" podID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerID="e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894" exitCode=0 Oct 30 18:42:45.472888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.472797 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" event={"ID":"757179a3-982f-402c-8b8a-00acf2cc8f56","Type":"ContainerDied","Data":"e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894"} Oct 30 18:42:45.473150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.473136 4862 scope.go:122] "RemoveContainer" containerID="e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894" Oct 30 18:42:45.475338 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.475310 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:45.475435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.475379 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:45.475435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.475416 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:45.510599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:45.510532 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" podStartSLOduration=2.510518673 podStartE2EDuration="2.510518673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:45.508896843 +0000 UTC m=+2664.557941757" watchObservedRunningTime="2026-10-30 18:42:45.510518673 +0000 UTC m=+2664.559563586" Oct 30 18:42:46.481847 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:46.481811 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" event={"ID":"757179a3-982f-402c-8b8a-00acf2cc8f56","Type":"ContainerStarted","Data":"c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2"} Oct 30 18:42:46.541615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:46.541586 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:42:46.541768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:46.541692 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:42:46.541768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:46.541730 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:42:47.550285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:47.549722 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:42:47.550285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:47.549780 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:42:48.551288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:48.550574 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:42:48.551288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:48.550632 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:42:49.554660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:49.554613 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:42:49.555139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:49.554688 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:42:49.879334 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:42:49.879223 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:42:50.548292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:50.541592 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:42:50.548292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:50.541669 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:42:51.553413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:51.553370 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:42:51.553911 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:51.553429 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:42:52.546137 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:52.546109 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:42:52.546519 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:52.546495 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:42:52.547183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:52.547167 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 18:42:55.725052 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:55.721403 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:42:56.486245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:42:56.486214 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:43:00.344606 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.344574 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57"] Oct 30 18:43:00.371525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.371489 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57"] Oct 30 18:43:00.371710 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.371625 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.388594 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.388571 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:43:00.388961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.388942 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:43:00.405186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.405154 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744a0ac7-vpt57\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.405357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.405337 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744a0ac7-vpt57\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.405467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.405450 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spk8d\" (UniqueName: \"kubernetes.io/projected/2d57ae37-021a-452b-a7d9-ee659651a258-kube-api-access-spk8d\") pod \"maas-api-revoke-sub-744a0ac7-vpt57\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.506985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.506956 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744a0ac7-vpt57\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.507159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.507070 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spk8d\" (UniqueName: \"kubernetes.io/projected/2d57ae37-021a-452b-a7d9-ee659651a258-kube-api-access-spk8d\") pod \"maas-api-revoke-sub-744a0ac7-vpt57\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.507159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.507153 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744a0ac7-vpt57\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.507846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.507790 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744a0ac7-vpt57\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.509567 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.509546 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744a0ac7-vpt57\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.526415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.526388 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spk8d\" (UniqueName: \"kubernetes.io/projected/2d57ae37-021a-452b-a7d9-ee659651a258-kube-api-access-spk8d\") pod \"maas-api-revoke-sub-744a0ac7-vpt57\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:00.687680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:00.687604 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:01.013907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:01.013876 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57"] Oct 30 18:43:01.016023 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:43:01.015995 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d57ae37_021a_452b_a7d9_ee659651a258.slice/crio-ae312bc3442d95a899e8eb2da7d589fe70520b2529848073f81e28f3567db609 WatchSource:0}: Error finding container ae312bc3442d95a899e8eb2da7d589fe70520b2529848073f81e28f3567db609: Status 404 returned error can't find the container with id ae312bc3442d95a899e8eb2da7d589fe70520b2529848073f81e28f3567db609 Oct 30 18:43:01.616637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:01.616606 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" event={"ID":"2d57ae37-021a-452b-a7d9-ee659651a258","Type":"ContainerStarted","Data":"c8bac789017b08201d9691aea8d86d9f1abf6ac63cc463d2cd9dd2e6ab2e98c0"} Oct 30 18:43:01.617036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:01.616664 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" event={"ID":"2d57ae37-021a-452b-a7d9-ee659651a258","Type":"ContainerStarted","Data":"ae312bc3442d95a899e8eb2da7d589fe70520b2529848073f81e28f3567db609"} Oct 30 18:43:01.664377 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:01.664280 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" podStartSLOduration=1.664250006 podStartE2EDuration="1.664250006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:01.662939833 +0000 UTC m=+2680.711984758" watchObservedRunningTime="2026-10-30 18:43:01.664250006 +0000 UTC m=+2680.713294919" Oct 30 18:43:02.627882 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:02.627847 4862 generic.go:356] "Generic (PLEG): container finished" podID="2d57ae37-021a-452b-a7d9-ee659651a258" containerID="c8bac789017b08201d9691aea8d86d9f1abf6ac63cc463d2cd9dd2e6ab2e98c0" exitCode=0 Oct 30 18:43:02.628239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:02.627900 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" event={"ID":"2d57ae37-021a-452b-a7d9-ee659651a258","Type":"ContainerDied","Data":"c8bac789017b08201d9691aea8d86d9f1abf6ac63cc463d2cd9dd2e6ab2e98c0"} Oct 30 18:43:03.804236 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.804207 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:03.886846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.886809 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-openshift-ca\") pod \"2d57ae37-021a-452b-a7d9-ee659651a258\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " Oct 30 18:43:03.887101 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.886891 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-xks-ca\") pod \"2d57ae37-021a-452b-a7d9-ee659651a258\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " Oct 30 18:43:03.887101 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.886985 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d57ae37-021a-452b-a7d9-ee659651a258-kube-api-access-spk8d\" (UniqueName: \"kubernetes.io/projected/2d57ae37-021a-452b-a7d9-ee659651a258-kube-api-access-spk8d\") pod \"2d57ae37-021a-452b-a7d9-ee659651a258\" (UID: \"2d57ae37-021a-452b-a7d9-ee659651a258\") " Oct 30 18:43:03.888831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.888685 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-openshift-ca" pod "2d57ae37-021a-452b-a7d9-ee659651a258" (UID: "2d57ae37-021a-452b-a7d9-ee659651a258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:03.890630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.890599 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d57ae37-021a-452b-a7d9-ee659651a258-kube-api-access-spk8d" pod "2d57ae37-021a-452b-a7d9-ee659651a258" (UID: "2d57ae37-021a-452b-a7d9-ee659651a258"). InnerVolumeSpecName "kube-api-access-spk8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:03.891322 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.891230 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-xks-ca" pod "2d57ae37-021a-452b-a7d9-ee659651a258" (UID: "2d57ae37-021a-452b-a7d9-ee659651a258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:03.988411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.988379 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spk8d\" (UniqueName: \"kubernetes.io/projected/2d57ae37-021a-452b-a7d9-ee659651a258-kube-api-access-spk8d\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:03.988411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.988410 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:03.988614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:03.988424 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d57ae37-021a-452b-a7d9-ee659651a258-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:04.644547 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:04.644509 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" event={"ID":"2d57ae37-021a-452b-a7d9-ee659651a258","Type":"ContainerDied","Data":"ae312bc3442d95a899e8eb2da7d589fe70520b2529848073f81e28f3567db609"} Oct 30 18:43:04.644547 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:04.644539 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57" Oct 30 18:43:04.644766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:04.644573 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae312bc3442d95a899e8eb2da7d589fe70520b2529848073f81e28f3567db609" Oct 30 18:43:05.659061 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:05.659027 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm"] Oct 30 18:43:05.659672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:05.659303 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" containerID="cri-o://764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b" gracePeriod=30 Oct 30 18:43:06.666393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:06.666354 4862 generic.go:356] "Generic (PLEG): container finished" podID="6a741c85-d6fc-4bac-82c1-58884544c340" containerID="366f11bbdc5f36eebbeabbe5400169b0b9c58a78da49e93a27eac5726b9c54ff" exitCode=0 Oct 30 18:43:06.666926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:06.666440 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" event={"ID":"6a741c85-d6fc-4bac-82c1-58884544c340","Type":"ContainerDied","Data":"366f11bbdc5f36eebbeabbe5400169b0b9c58a78da49e93a27eac5726b9c54ff"} Oct 30 18:43:06.666926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:06.666821 4862 scope.go:122] "RemoveContainer" containerID="366f11bbdc5f36eebbeabbe5400169b0b9c58a78da49e93a27eac5726b9c54ff" Oct 30 18:43:06.668604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:06.668580 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:43:06.668669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:06.668649 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:43:06.668712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:06.668682 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:43:07.675918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:07.675885 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" event={"ID":"6a741c85-d6fc-4bac-82c1-58884544c340","Type":"ContainerStarted","Data":"a6c60e5fe5a3c611d3b889daee851dd785c0ed9bcefc566084452aac445c4fa1"} Oct 30 18:43:08.510609 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:08.510579 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:43:08.512289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:08.512235 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:43:08.512442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:08.512301 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 18:43:09.510809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:09.510763 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:43:09.511327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:09.510832 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 18:43:10.510679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:10.510647 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:43:10.510873 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:10.510717 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 18:43:10.987776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:10.987733 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:43:11.008168 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.008148 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:43:11.102675 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.102645 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-socket\") pod \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " Oct 30 18:43:11.102868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.102842 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-data\") pod \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " Oct 30 18:43:11.102959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.102940 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-podinfo\") pod \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " Oct 30 18:43:11.103000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.102957 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-socket" pod "a0f3e3b7-d1d3-49fb-8106-a26764c54660" (UID: "a0f3e3b7-d1d3-49fb-8106-a26764c54660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:11.103045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.103008 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-certs\") pod \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " Oct 30 18:43:11.103113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.103093 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-envoy\") pod \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " Oct 30 18:43:11.103376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.103355 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-kube-api-access-jh9qd\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-kube-api-access-jh9qd\") pod \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " Oct 30 18:43:11.103567 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.103547 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-token\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-token\") pod \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " Oct 30 18:43:11.103729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.103714 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istiod-ca-cert\") pod \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " Oct 30 18:43:11.103896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.103880 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-credential-socket\") pod \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\" (UID: \"a0f3e3b7-d1d3-49fb-8106-a26764c54660\") " Oct 30 18:43:11.104180 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.104134 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-certs" pod "a0f3e3b7-d1d3-49fb-8106-a26764c54660" (UID: "a0f3e3b7-d1d3-49fb-8106-a26764c54660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:11.104292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.104276 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istiod-ca-cert" pod "a0f3e3b7-d1d3-49fb-8106-a26764c54660" (UID: "a0f3e3b7-d1d3-49fb-8106-a26764c54660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:11.104389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.104368 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-data" pod "a0f3e3b7-d1d3-49fb-8106-a26764c54660" (UID: "a0f3e3b7-d1d3-49fb-8106-a26764c54660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:11.104623 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.104551 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-credential-socket" pod "a0f3e3b7-d1d3-49fb-8106-a26764c54660" (UID: "a0f3e3b7-d1d3-49fb-8106-a26764c54660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:11.105535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.105148 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:11.105535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.105173 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:11.105535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.105189 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:11.105535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.105206 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:11.105535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.105220 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:11.107133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.107039 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-kube-api-access-jh9qd" pod "a0f3e3b7-d1d3-49fb-8106-a26764c54660" (UID: "a0f3e3b7-d1d3-49fb-8106-a26764c54660"). InnerVolumeSpecName "kube-api-access-jh9qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:11.107133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.107068 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-podinfo" pod "a0f3e3b7-d1d3-49fb-8106-a26764c54660" (UID: "a0f3e3b7-d1d3-49fb-8106-a26764c54660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:43:11.107133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.107116 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-token" pod "a0f3e3b7-d1d3-49fb-8106-a26764c54660" (UID: "a0f3e3b7-d1d3-49fb-8106-a26764c54660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:11.108320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.108281 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-envoy" pod "a0f3e3b7-d1d3-49fb-8106-a26764c54660" (UID: "a0f3e3b7-d1d3-49fb-8106-a26764c54660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:11.206831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.206805 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:11.206831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.206831 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:11.207023 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.206844 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh9qd\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-kube-api-access-jh9qd\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:11.207023 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.206857 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0f3e3b7-d1d3-49fb-8106-a26764c54660-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:11.482712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.482624 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j"] Oct 30 18:43:11.512217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.510452 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:43:11.512217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.510594 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 18:43:11.704323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.704254 4862 generic.go:356] "Generic (PLEG): container finished" podID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerID="764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b" exitCode=0 Oct 30 18:43:11.704323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.704321 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" Oct 30 18:43:11.704588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.704332 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" event={"ID":"a0f3e3b7-d1d3-49fb-8106-a26764c54660","Type":"ContainerDied","Data":"764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b"} Oct 30 18:43:11.704588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.704394 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm" event={"ID":"a0f3e3b7-d1d3-49fb-8106-a26764c54660","Type":"ContainerDied","Data":"6a21c1ef6c246d8890f1417d018d132d9086b483f5801e404e37925ab8274638"} Oct 30 18:43:11.704588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.704425 4862 scope.go:122] "RemoveContainer" containerID="764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b" Oct 30 18:43:11.715371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.715356 4862 scope.go:122] "RemoveContainer" containerID="764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b" Oct 30 18:43:11.715669 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:43:11.715647 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b\": container with ID starting with 764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b not found: ID does not exist" containerID="764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b" Oct 30 18:43:11.715742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.715681 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b"} err="failed to get container status \"764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b\": rpc error: code = NotFound desc = could not find container \"764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b\": container with ID starting with 764ce07ff747d972c2963df4c961832929a174f70716f1547953a08e30c50c4b not found: ID does not exist" Oct 30 18:43:11.773495 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.773428 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm"] Oct 30 18:43:11.778854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:11.778835 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-ab244c-openshift-default-694869f597-h76qm"] Oct 30 18:43:12.510383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:12.510355 4862 patch_prober.go:29] interesting pod/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:43:12.510635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:12.510426 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 18:43:13.064136 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:13.064085 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" path="/var/lib/kubelet/pods/a0f3e3b7-d1d3-49fb-8106-a26764c54660/volumes" Oct 30 18:43:13.515102 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:13.515035 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:43:13.516006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:13.515986 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 18:43:14.027559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:14.027526 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq"] Oct 30 18:43:14.030287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:14.030253 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2263df1-wk2nq"] Oct 30 18:43:15.063148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:15.063112 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b707dba-6650-4aa6-8133-28bbfddc5dae" path="/var/lib/kubelet/pods/8b707dba-6650-4aa6-8133-28bbfddc5dae/volumes" Oct 30 18:43:15.083302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:15.083250 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn"] Oct 30 18:43:15.603297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:15.603251 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk"] Oct 30 18:43:15.603491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:15.603449 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" containerID="cri-o://4da1817d2f86a529f3eda24b4a9f039ce57c626d4127befa855052d1f1b165b0" gracePeriod=30 Oct 30 18:43:16.483301 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.483227 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" podUID="91feb246-58aa-448b-9d12-9b97d7589271" containerName="payload-processing" containerID="cri-o://76cd77cfc76d6c9e5613ffbba71ec21320cf3db2ac99d35a2bd238d791c416b9" gracePeriod=25 Oct 30 18:43:16.749889 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.749826 4862 generic.go:356] "Generic (PLEG): container finished" podID="91feb246-58aa-448b-9d12-9b97d7589271" containerID="76cd77cfc76d6c9e5613ffbba71ec21320cf3db2ac99d35a2bd238d791c416b9" exitCode=0 Oct 30 18:43:16.750016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.749894 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" event={"ID":"91feb246-58aa-448b-9d12-9b97d7589271","Type":"ContainerDied","Data":"76cd77cfc76d6c9e5613ffbba71ec21320cf3db2ac99d35a2bd238d791c416b9"} Oct 30 18:43:16.750016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.749942 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" event={"ID":"91feb246-58aa-448b-9d12-9b97d7589271","Type":"ContainerDied","Data":"20d6e6d78336dc0493d8099ab3b3d16b2b955017b65b6a2b1f13b875819c0d17"} Oct 30 18:43:16.750016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.749958 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d6e6d78336dc0493d8099ab3b3d16b2b955017b65b6a2b1f13b875819c0d17" Oct 30 18:43:16.755853 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.755837 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:43:16.815773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.815746 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-provider-credentials\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-provider-credentials\") pod \"91feb246-58aa-448b-9d12-9b97d7589271\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " Oct 30 18:43:16.815937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.815894 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-routing-overlay-volume\") pod \"91feb246-58aa-448b-9d12-9b97d7589271\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " Oct 30 18:43:16.816005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.815936 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-plugins-config-volume\") pod \"91feb246-58aa-448b-9d12-9b97d7589271\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " Oct 30 18:43:16.816005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.815999 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-kube-api-access-zwqdv\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-kube-api-access-zwqdv\") pod \"91feb246-58aa-448b-9d12-9b97d7589271\" (UID: \"91feb246-58aa-448b-9d12-9b97d7589271\") " Oct 30 18:43:16.816371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.816251 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-routing-overlay-volume" pod "91feb246-58aa-448b-9d12-9b97d7589271" (UID: "91feb246-58aa-448b-9d12-9b97d7589271"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:16.816698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.816493 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-plugins-config-volume" pod "91feb246-58aa-448b-9d12-9b97d7589271" (UID: "91feb246-58aa-448b-9d12-9b97d7589271"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:16.818241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.818220 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-provider-credentials" pod "91feb246-58aa-448b-9d12-9b97d7589271" (UID: "91feb246-58aa-448b-9d12-9b97d7589271"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:16.818385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.818361 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-kube-api-access-zwqdv" pod "91feb246-58aa-448b-9d12-9b97d7589271" (UID: "91feb246-58aa-448b-9d12-9b97d7589271"). InnerVolumeSpecName "kube-api-access-zwqdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:16.917755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.917727 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:16.917755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.917750 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:16.917755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.917760 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91feb246-58aa-448b-9d12-9b97d7589271-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:16.917946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:16.917770 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqdv\" (UniqueName: \"kubernetes.io/projected/91feb246-58aa-448b-9d12-9b97d7589271-kube-api-access-zwqdv\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:17.755844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:17.755817 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j" Oct 30 18:43:17.807354 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:17.807324 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j"] Oct 30 18:43:17.819526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:17.819499 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7ccdmsq5j"] Oct 30 18:43:19.062890 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:19.062862 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91feb246-58aa-448b-9d12-9b97d7589271" path="/var/lib/kubelet/pods/91feb246-58aa-448b-9d12-9b97d7589271/volumes" Oct 30 18:43:19.870546 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:43:19.870512 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:43:20.084406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.084372 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" podUID="fdd083e4-1455-4a3e-856f-0b7947d931e4" containerName="payload-processing" containerID="cri-o://29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952" gracePeriod=25 Oct 30 18:43:20.302796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.302769 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj"] Oct 30 18:43:20.303000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.302976 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" containerID="cri-o://40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a" gracePeriod=30 Oct 30 18:43:20.390289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.390238 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:43:20.469440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.469373 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-routing-overlay-volume\") pod \"fdd083e4-1455-4a3e-856f-0b7947d931e4\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " Oct 30 18:43:20.469590 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.469491 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-plugins-config-volume\") pod \"fdd083e4-1455-4a3e-856f-0b7947d931e4\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " Oct 30 18:43:20.469638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.469592 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-provider-credentials\") pod \"fdd083e4-1455-4a3e-856f-0b7947d931e4\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " Oct 30 18:43:20.469638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.469630 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-kube-api-access-rz4pc\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-kube-api-access-rz4pc\") pod \"fdd083e4-1455-4a3e-856f-0b7947d931e4\" (UID: \"fdd083e4-1455-4a3e-856f-0b7947d931e4\") " Oct 30 18:43:20.469987 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.469814 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-routing-overlay-volume" pod "fdd083e4-1455-4a3e-856f-0b7947d931e4" (UID: "fdd083e4-1455-4a3e-856f-0b7947d931e4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:20.469987 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.469957 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-plugins-config-volume" pod "fdd083e4-1455-4a3e-856f-0b7947d931e4" (UID: "fdd083e4-1455-4a3e-856f-0b7947d931e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:20.470185 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.470166 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:20.470229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.470192 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdd083e4-1455-4a3e-856f-0b7947d931e4-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:20.472236 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.472215 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-provider-credentials" pod "fdd083e4-1455-4a3e-856f-0b7947d931e4" (UID: "fdd083e4-1455-4a3e-856f-0b7947d931e4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:20.472529 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.472501 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-kube-api-access-rz4pc" pod "fdd083e4-1455-4a3e-856f-0b7947d931e4" (UID: "fdd083e4-1455-4a3e-856f-0b7947d931e4"). InnerVolumeSpecName "kube-api-access-rz4pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:20.577519 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.574317 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:20.577519 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.574355 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4pc\" (UniqueName: \"kubernetes.io/projected/fdd083e4-1455-4a3e-856f-0b7947d931e4-kube-api-access-rz4pc\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:20.779947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.779903 4862 generic.go:356] "Generic (PLEG): container finished" podID="fdd083e4-1455-4a3e-856f-0b7947d931e4" containerID="29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952" exitCode=0 Oct 30 18:43:20.780127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.780039 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" event={"ID":"fdd083e4-1455-4a3e-856f-0b7947d931e4","Type":"ContainerDied","Data":"29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952"} Oct 30 18:43:20.780127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.780124 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" event={"ID":"fdd083e4-1455-4a3e-856f-0b7947d931e4","Type":"ContainerDied","Data":"cafce3fd5e55348b9ff1f7c799928e34be6078969212bea74568958dbcc3ce61"} Oct 30 18:43:20.780316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.780154 4862 scope.go:122] "RemoveContainer" containerID="29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952" Oct 30 18:43:20.780383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.780352 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn" Oct 30 18:43:20.784539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.784510 4862 generic.go:356] "Generic (PLEG): container finished" podID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerID="4da1817d2f86a529f3eda24b4a9f039ce57c626d4127befa855052d1f1b165b0" exitCode=0 Oct 30 18:43:20.784642 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.784538 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" event={"ID":"9fd56d44-f570-45f5-9e1a-c3de0ff49b11","Type":"ContainerDied","Data":"4da1817d2f86a529f3eda24b4a9f039ce57c626d4127befa855052d1f1b165b0"} Oct 30 18:43:20.803497 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.803464 4862 scope.go:122] "RemoveContainer" containerID="29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952" Oct 30 18:43:20.807852 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:43:20.807813 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952\": container with ID starting with 29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952 not found: ID does not exist" containerID="29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952" Oct 30 18:43:20.807963 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.807854 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952"} err="failed to get container status \"29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952\": rpc error: code = NotFound desc = could not find container \"29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952\": container with ID starting with 29e9a8e7f402e237ad0c17d24cfc1f32d3b342f7790cf45f42d69c2106ebe952 not found: ID does not exist" Oct 30 18:43:20.845361 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.845325 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn"] Oct 30 18:43:20.847302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.847254 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679nwkcn"] Oct 30 18:43:20.871210 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.871193 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:43:20.980885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.980808 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-kube-api-access-5fjxt\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-kube-api-access-5fjxt\") pod \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " Oct 30 18:43:20.980885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.980857 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-token\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-token\") pod \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " Oct 30 18:43:20.980885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.980887 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-socket\") pod \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " Oct 30 18:43:20.981171 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.980953 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-podinfo\") pod \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " Oct 30 18:43:20.981171 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.980994 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istiod-ca-cert\") pod \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " Oct 30 18:43:20.981171 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981021 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-certs\") pod \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " Oct 30 18:43:20.981171 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981128 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-data\") pod \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " Oct 30 18:43:20.981427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981290 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-envoy\") pod \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " Oct 30 18:43:20.981427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981248 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-socket" pod "9fd56d44-f570-45f5-9e1a-c3de0ff49b11" (UID: "9fd56d44-f570-45f5-9e1a-c3de0ff49b11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:20.981427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981332 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-credential-socket\") pod \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\" (UID: \"9fd56d44-f570-45f5-9e1a-c3de0ff49b11\") " Oct 30 18:43:20.981427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981394 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-certs" pod "9fd56d44-f570-45f5-9e1a-c3de0ff49b11" (UID: "9fd56d44-f570-45f5-9e1a-c3de0ff49b11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:20.981836 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981525 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-data" pod "9fd56d44-f570-45f5-9e1a-c3de0ff49b11" (UID: "9fd56d44-f570-45f5-9e1a-c3de0ff49b11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:20.981836 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981551 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istiod-ca-cert" pod "9fd56d44-f570-45f5-9e1a-c3de0ff49b11" (UID: "9fd56d44-f570-45f5-9e1a-c3de0ff49b11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:20.981836 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981785 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-credential-socket" pod "9fd56d44-f570-45f5-9e1a-c3de0ff49b11" (UID: "9fd56d44-f570-45f5-9e1a-c3de0ff49b11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:20.982227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.981967 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:20.982227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.982053 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:20.982227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.982166 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:20.982227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.982180 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:20.982227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.982194 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:20.983747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.983726 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-token" pod "9fd56d44-f570-45f5-9e1a-c3de0ff49b11" (UID: "9fd56d44-f570-45f5-9e1a-c3de0ff49b11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:20.983747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.983739 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-kube-api-access-5fjxt" pod "9fd56d44-f570-45f5-9e1a-c3de0ff49b11" (UID: "9fd56d44-f570-45f5-9e1a-c3de0ff49b11"). InnerVolumeSpecName "kube-api-access-5fjxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:20.983959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.983934 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-podinfo" pod "9fd56d44-f570-45f5-9e1a-c3de0ff49b11" (UID: "9fd56d44-f570-45f5-9e1a-c3de0ff49b11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:43:20.984073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:20.984058 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-envoy" pod "9fd56d44-f570-45f5-9e1a-c3de0ff49b11" (UID: "9fd56d44-f570-45f5-9e1a-c3de0ff49b11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:21.064851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.064822 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdd083e4-1455-4a3e-856f-0b7947d931e4" path="/var/lib/kubelet/pods/fdd083e4-1455-4a3e-856f-0b7947d931e4/volumes" Oct 30 18:43:21.082760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.082736 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:21.082760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.082760 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:21.082915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.082769 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:21.082915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.082778 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fjxt\" (UniqueName: \"kubernetes.io/projected/9fd56d44-f570-45f5-9e1a-c3de0ff49b11-kube-api-access-5fjxt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:21.580277 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.580234 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:43:21.580713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.580425 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:43:21.793577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.793545 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" event={"ID":"9fd56d44-f570-45f5-9e1a-c3de0ff49b11","Type":"ContainerDied","Data":"05b1256a87f2e94f73066dc6e4391d7d428841a2a1699815874218d4dec71720"} Oct 30 18:43:21.793577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.793575 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk" Oct 30 18:43:21.793770 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.793607 4862 scope.go:122] "RemoveContainer" containerID="4da1817d2f86a529f3eda24b4a9f039ce57c626d4127befa855052d1f1b165b0" Oct 30 18:43:21.858851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.858781 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk"] Oct 30 18:43:21.866753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.866727 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1a2436-openshift-default-84b85bb969-6swpk"] Oct 30 18:43:21.899878 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:21.899855 4862 scope.go:122] "RemoveContainer" containerID="c56a3dab1d7f69a775404f6f97ac3862907ac8be0a1e204f0fed6357426e1749" Oct 30 18:43:23.065456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:23.063123 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" path="/var/lib/kubelet/pods/9fd56d44-f570-45f5-9e1a-c3de0ff49b11/volumes" Oct 30 18:43:25.587744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.587681 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:43:25.744326 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744298 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-podinfo\") pod \"30d6d3ce-dafd-4128-8805-382270e9f63f\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " Oct 30 18:43:25.744501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744378 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d6d3ce-dafd-4128-8805-382270e9f63f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6d3ce-dafd-4128-8805-382270e9f63f-istiod-ca-cert\") pod \"30d6d3ce-dafd-4128-8805-382270e9f63f\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " Oct 30 18:43:25.744501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744413 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-certs\") pod \"30d6d3ce-dafd-4128-8805-382270e9f63f\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " Oct 30 18:43:25.744501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744445 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-token\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-token\") pod \"30d6d3ce-dafd-4128-8805-382270e9f63f\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " Oct 30 18:43:25.744501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744468 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-credential-socket\") pod \"30d6d3ce-dafd-4128-8805-382270e9f63f\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " Oct 30 18:43:25.744692 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744547 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-envoy\") pod \"30d6d3ce-dafd-4128-8805-382270e9f63f\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " Oct 30 18:43:25.744692 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744625 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-socket\") pod \"30d6d3ce-dafd-4128-8805-382270e9f63f\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " Oct 30 18:43:25.744692 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744658 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-data\") pod \"30d6d3ce-dafd-4128-8805-382270e9f63f\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " Oct 30 18:43:25.744943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744699 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-kube-api-access-mz9k5\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-kube-api-access-mz9k5\") pod \"30d6d3ce-dafd-4128-8805-382270e9f63f\" (UID: \"30d6d3ce-dafd-4128-8805-382270e9f63f\") " Oct 30 18:43:25.744943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744702 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-certs" pod "30d6d3ce-dafd-4128-8805-382270e9f63f" (UID: "30d6d3ce-dafd-4128-8805-382270e9f63f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:25.744943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744803 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-credential-socket" pod "30d6d3ce-dafd-4128-8805-382270e9f63f" (UID: "30d6d3ce-dafd-4128-8805-382270e9f63f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:25.744943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.744813 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d6d3ce-dafd-4128-8805-382270e9f63f-istiod-ca-cert" pod "30d6d3ce-dafd-4128-8805-382270e9f63f" (UID: "30d6d3ce-dafd-4128-8805-382270e9f63f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:25.745287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.745033 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-socket" pod "30d6d3ce-dafd-4128-8805-382270e9f63f" (UID: "30d6d3ce-dafd-4128-8805-382270e9f63f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:25.745287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.745042 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-data" pod "30d6d3ce-dafd-4128-8805-382270e9f63f" (UID: "30d6d3ce-dafd-4128-8805-382270e9f63f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:25.745581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.745508 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.745581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.745529 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.745581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.745544 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30d6d3ce-dafd-4128-8805-382270e9f63f-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.745581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.745558 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.745581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.745571 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.746810 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.746771 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-podinfo" pod "30d6d3ce-dafd-4128-8805-382270e9f63f" (UID: "30d6d3ce-dafd-4128-8805-382270e9f63f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:43:25.747021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.747002 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-token" pod "30d6d3ce-dafd-4128-8805-382270e9f63f" (UID: "30d6d3ce-dafd-4128-8805-382270e9f63f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:25.747108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.747032 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-kube-api-access-mz9k5" pod "30d6d3ce-dafd-4128-8805-382270e9f63f" (UID: "30d6d3ce-dafd-4128-8805-382270e9f63f"). InnerVolumeSpecName "kube-api-access-mz9k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:25.747163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.747127 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-envoy" pod "30d6d3ce-dafd-4128-8805-382270e9f63f" (UID: "30d6d3ce-dafd-4128-8805-382270e9f63f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:43:25.818404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.818364 4862 generic.go:356] "Generic (PLEG): container finished" podID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerID="40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a" exitCode=0 Oct 30 18:43:25.818582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.818431 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" Oct 30 18:43:25.818582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.818445 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" event={"ID":"30d6d3ce-dafd-4128-8805-382270e9f63f","Type":"ContainerDied","Data":"40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a"} Oct 30 18:43:25.818582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.818504 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj" event={"ID":"30d6d3ce-dafd-4128-8805-382270e9f63f","Type":"ContainerDied","Data":"053a03caa93d4a403f0a198c26d7fd0022c409dafb99f459d5aca1110e55815e"} Oct 30 18:43:25.818582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.818525 4862 scope.go:122] "RemoveContainer" containerID="40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a" Oct 30 18:43:25.830483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.830464 4862 scope.go:122] "RemoveContainer" containerID="40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a" Oct 30 18:43:25.830761 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:43:25.830733 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a\": container with ID starting with 40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a not found: ID does not exist" containerID="40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a" Oct 30 18:43:25.830863 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.830768 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a"} err="failed to get container status \"40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a\": rpc error: code = NotFound desc = could not find container \"40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a\": container with ID starting with 40cc00a6627b632695210f0fc8563ce0b5bf5309708f4b6832ac4ab22d9f6f9a not found: ID does not exist" Oct 30 18:43:25.847008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.846963 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.847008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.846983 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.847008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.846992 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30d6d3ce-dafd-4128-8805-382270e9f63f-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.847008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.847001 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz9k5\" (UniqueName: \"kubernetes.io/projected/30d6d3ce-dafd-4128-8805-382270e9f63f-kube-api-access-mz9k5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:25.872682 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.872658 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj"] Oct 30 18:43:25.879551 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.879529 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4a7508-openshift-default-546fcc75b9-66rmj"] Oct 30 18:43:25.887245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887226 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2"] Oct 30 18:43:25.887765 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887751 4862 cpu_manager.go:414] "containerMap: removing container" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" Oct 30 18:43:25.887815 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887766 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" Oct 30 18:43:25.887815 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887792 4862 cpu_manager.go:414] "containerMap: removing container" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" Oct 30 18:43:25.887815 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887797 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" Oct 30 18:43:25.887815 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887804 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2d57ae37-021a-452b-a7d9-ee659651a258" containerName="revoke-keys" Oct 30 18:43:25.887815 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887809 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d57ae37-021a-452b-a7d9-ee659651a258" containerName="revoke-keys" Oct 30 18:43:25.887984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887818 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" Oct 30 18:43:25.887984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887823 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" Oct 30 18:43:25.887984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887835 4862 cpu_manager.go:414] "containerMap: removing container" podUID="fdd083e4-1455-4a3e-856f-0b7947d931e4" containerName="payload-processing" Oct 30 18:43:25.887984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887839 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdd083e4-1455-4a3e-856f-0b7947d931e4" containerName="payload-processing" Oct 30 18:43:25.887984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887857 4862 cpu_manager.go:414] "containerMap: removing container" podUID="91feb246-58aa-448b-9d12-9b97d7589271" containerName="payload-processing" Oct 30 18:43:25.887984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887861 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91feb246-58aa-448b-9d12-9b97d7589271" containerName="payload-processing" Oct 30 18:43:25.887984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887966 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd56d44-f570-45f5-9e1a-c3de0ff49b11" containerName="istio-proxy" Oct 30 18:43:25.887984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887974 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d57ae37-021a-452b-a7d9-ee659651a258" containerName="revoke-keys" Oct 30 18:43:25.887984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887982 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f3e3b7-d1d3-49fb-8106-a26764c54660" containerName="istio-proxy" Oct 30 18:43:25.888303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.887991 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdd083e4-1455-4a3e-856f-0b7947d931e4" containerName="payload-processing" Oct 30 18:43:25.888303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.888000 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="91feb246-58aa-448b-9d12-9b97d7589271" containerName="payload-processing" Oct 30 18:43:25.888303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.888007 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" containerName="istio-proxy" Oct 30 18:43:25.892758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.892742 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:25.907708 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.907687 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2107d4\"" Oct 30 18:43:25.907844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.907690 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"openshift-service-ca.crt\"" Oct 30 18:43:25.907844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.907687 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"routing-overlay\"" Oct 30 18:43:25.907969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.907847 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"e2e-cmp-legacy-f3d4ec-provider-cred\"" Oct 30 18:43:25.907969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.907862 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"kube-root-ca.crt\"" Oct 30 18:43:25.907969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.907877 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"payload-processing-external-model-e2e--41b9a229-dockercfg-r8xmm\"" Oct 30 18:43:25.908596 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:25.908578 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2"] Oct 30 18:43:26.049992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.049545 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.049992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.049689 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.049992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.049761 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqzq\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-kube-api-access-sxqzq\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.049992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.049880 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.150661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.150578 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.150661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.150639 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.150936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.150780 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.150936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.150877 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqzq\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-kube-api-access-sxqzq\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.151393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.151370 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.152091 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.152065 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.153588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.153568 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.170809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.170788 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqzq\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-kube-api-access-sxqzq\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.208283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.207220 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.533983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.533952 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2"] Oct 30 18:43:26.535309 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:43:26.535285 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288f283c_eff8_4ade_9b4b_6543280108a3.slice/crio-c99ad38f9a2a800f43d0e6c0836d0bf3b609d5226202d19c06d070daf2c5c984 WatchSource:0}: Error finding container c99ad38f9a2a800f43d0e6c0836d0bf3b609d5226202d19c06d070daf2c5c984: Status 404 returned error can't find the container with id c99ad38f9a2a800f43d0e6c0836d0bf3b609d5226202d19c06d070daf2c5c984 Oct 30 18:43:26.827395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.827355 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" event={"ID":"288f283c-eff8-4ade-9b4b-6543280108a3","Type":"ContainerStarted","Data":"f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8"} Oct 30 18:43:26.827848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.827445 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" event={"ID":"288f283c-eff8-4ade-9b4b-6543280108a3","Type":"ContainerStarted","Data":"c99ad38f9a2a800f43d0e6c0836d0bf3b609d5226202d19c06d070daf2c5c984"} Oct 30 18:43:26.827848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.827592 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:26.867505 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:26.867457 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" podStartSLOduration=1.867444156 podStartE2EDuration="1.867444156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:26.866142802 +0000 UTC m=+2705.915187715" watchObservedRunningTime="2026-10-30 18:43:26.867444156 +0000 UTC m=+2705.916489066" Oct 30 18:43:27.063045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:27.063011 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d6d3ce-dafd-4128-8805-382270e9f63f" path="/var/lib/kubelet/pods/30d6d3ce-dafd-4128-8805-382270e9f63f/volumes" Oct 30 18:43:32.884388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.884349 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg"] Oct 30 18:43:32.889907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.889884 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:32.906624 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.906599 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg"] Oct 30 18:43:32.916208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.916078 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"kube-root-ca.crt\"" Oct 30 18:43:32.916208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.916136 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"openshift-service-ca.crt\"" Oct 30 18:43:32.916208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.916200 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8b8e54\"" Oct 30 18:43:32.916895 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.916327 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"routing-overlay\"" Oct 30 18:43:32.916959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.916932 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"agc-cred-36f21024123d\"" Oct 30 18:43:32.917157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.917138 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"payload-processing-external-model-e2e--13a66b56-dockercfg-kjqxg\"" Oct 30 18:43:32.917228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.917187 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"agc-cred-33ba6407b31e\"" Oct 30 18:43:32.917228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:32.917220 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8b8e54\"/\"agc-cred-695bf65e6f2e\"" Oct 30 18:43:33.041983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.041954 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.042127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.042001 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.042127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.042114 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.042293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.042244 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncv69\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-kube-api-access-ncv69\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.143621 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.143556 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncv69\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-kube-api-access-ncv69\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.143621 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.143611 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.143794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.143643 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.143845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.143803 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.144451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.144425 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.144552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.144448 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.146413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.146397 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.162151 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.162127 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncv69\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-kube-api-access-ncv69\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.204760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.204740 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.506601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.506576 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg"] Oct 30 18:43:33.507815 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:43:33.507788 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9baac7_c92c_4564_b098_ac80fc122bf2.slice/crio-a7333802dfc5a99eaeaeb348061ba7ae61b13c42736943173e840ba7c1622060 WatchSource:0}: Error finding container a7333802dfc5a99eaeaeb348061ba7ae61b13c42736943173e840ba7c1622060: Status 404 returned error can't find the container with id a7333802dfc5a99eaeaeb348061ba7ae61b13c42736943173e840ba7c1622060 Oct 30 18:43:33.896579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.896541 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" event={"ID":"ee9baac7-c92c-4564-b098-ac80fc122bf2","Type":"ContainerStarted","Data":"b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43"} Oct 30 18:43:33.896989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.896606 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" event={"ID":"ee9baac7-c92c-4564-b098-ac80fc122bf2","Type":"ContainerStarted","Data":"a7333802dfc5a99eaeaeb348061ba7ae61b13c42736943173e840ba7c1622060"} Oct 30 18:43:33.896989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.896731 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:33.936498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:33.936457 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" podStartSLOduration=1.9364448969999999 podStartE2EDuration="1.936444897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:33.934966482 +0000 UTC m=+2712.984011394" watchObservedRunningTime="2026-10-30 18:43:33.936444897 +0000 UTC m=+2712.985489852" Oct 30 18:43:34.012288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.012236 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t"] Oct 30 18:43:34.016313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.016298 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.033110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.033083 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"kube-root-ca.crt\"" Oct 30 18:43:34.033249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.033239 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"agc-cred-b92e75ad3363\"" Oct 30 18:43:34.033374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.033363 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"routing-overlay\"" Oct 30 18:43:34.033470 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.033429 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"openshift-service-ca.crt\"" Oct 30 18:43:34.033578 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.033518 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1c4bcd\"" Oct 30 18:43:34.033578 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.033532 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"payload-processing-external-model-e2e--8a6e7c24-dockercfg-gfjch\"" Oct 30 18:43:34.036713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.036695 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t"] Oct 30 18:43:34.162341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.159062 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.162341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.159114 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg55\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-kube-api-access-shg55\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.162341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.159152 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.162341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.159900 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.260851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.260818 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.261019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.260929 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.261019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.260950 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg55\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-kube-api-access-shg55\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.261019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.260971 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.261630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.261603 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.261754 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.261689 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.263523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.263505 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.282961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.282939 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg55\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-kube-api-access-shg55\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.330550 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.330530 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.685651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.685628 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t"] Oct 30 18:43:34.686630 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:43:34.686604 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ea0d22_b1cd_4eca_916b_c5155ab1139c.slice/crio-4f4c858c27f2c1fb7a5757035437eb7c6b6a3f49b5958c7f6553f868884e91e4 WatchSource:0}: Error finding container 4f4c858c27f2c1fb7a5757035437eb7c6b6a3f49b5958c7f6553f868884e91e4: Status 404 returned error can't find the container with id 4f4c858c27f2c1fb7a5757035437eb7c6b6a3f49b5958c7f6553f868884e91e4 Oct 30 18:43:34.904715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.904680 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" event={"ID":"d3ea0d22-b1cd-4eca-916b-c5155ab1139c","Type":"ContainerStarted","Data":"3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120"} Oct 30 18:43:34.905124 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.904738 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" event={"ID":"d3ea0d22-b1cd-4eca-916b-c5155ab1139c","Type":"ContainerStarted","Data":"4f4c858c27f2c1fb7a5757035437eb7c6b6a3f49b5958c7f6553f868884e91e4"} Oct 30 18:43:34.905124 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.904894 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:34.946579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:34.946492 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" podStartSLOduration=1.946479396 podStartE2EDuration="1.946479396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:34.944257887 +0000 UTC m=+2713.993302812" watchObservedRunningTime="2026-10-30 18:43:34.946479396 +0000 UTC m=+2713.995524309" Oct 30 18:43:37.839565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:37.839534 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:43:39.488959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:39.488725 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t"] Oct 30 18:43:39.492464 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:39.492441 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:44.489286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.489224 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" podUID="d3ea0d22-b1cd-4eca-916b-c5155ab1139c" containerName="payload-processing" containerID="cri-o://3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120" gracePeriod=25 Oct 30 18:43:44.763189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.763162 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:44.894589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.894559 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-kube-api-access-shg55\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-kube-api-access-shg55\") pod \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " Oct 30 18:43:44.894772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.894598 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-provider-credentials\") pod \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " Oct 30 18:43:44.894772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.894666 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-routing-overlay-volume\") pod \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " Oct 30 18:43:44.894772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.894687 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-plugins-config-volume\") pod \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\" (UID: \"d3ea0d22-b1cd-4eca-916b-c5155ab1139c\") " Oct 30 18:43:44.895033 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.895005 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-routing-overlay-volume" pod "d3ea0d22-b1cd-4eca-916b-c5155ab1139c" (UID: "d3ea0d22-b1cd-4eca-916b-c5155ab1139c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:44.895145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.895123 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-plugins-config-volume" pod "d3ea0d22-b1cd-4eca-916b-c5155ab1139c" (UID: "d3ea0d22-b1cd-4eca-916b-c5155ab1139c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:44.895335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.895319 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:44.895395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.895340 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:44.897074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.897051 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-provider-credentials" pod "d3ea0d22-b1cd-4eca-916b-c5155ab1139c" (UID: "d3ea0d22-b1cd-4eca-916b-c5155ab1139c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:44.897164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.897103 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-kube-api-access-shg55" pod "d3ea0d22-b1cd-4eca-916b-c5155ab1139c" (UID: "d3ea0d22-b1cd-4eca-916b-c5155ab1139c"). InnerVolumeSpecName "kube-api-access-shg55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:44.908690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.908668 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:43:44.972216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.972185 4862 generic.go:356] "Generic (PLEG): container finished" podID="d3ea0d22-b1cd-4eca-916b-c5155ab1139c" containerID="3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120" exitCode=0 Oct 30 18:43:44.972402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.972232 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" Oct 30 18:43:44.972402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.972282 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" event={"ID":"d3ea0d22-b1cd-4eca-916b-c5155ab1139c","Type":"ContainerDied","Data":"3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120"} Oct 30 18:43:44.972402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.972335 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t" event={"ID":"d3ea0d22-b1cd-4eca-916b-c5155ab1139c","Type":"ContainerDied","Data":"4f4c858c27f2c1fb7a5757035437eb7c6b6a3f49b5958c7f6553f868884e91e4"} Oct 30 18:43:44.972402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.972363 4862 scope.go:122] "RemoveContainer" containerID="3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120" Oct 30 18:43:44.983706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.983682 4862 scope.go:122] "RemoveContainer" containerID="3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120" Oct 30 18:43:44.983998 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:43:44.983974 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120\": container with ID starting with 3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120 not found: ID does not exist" containerID="3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120" Oct 30 18:43:44.984057 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.984009 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120"} err="failed to get container status \"3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120\": rpc error: code = NotFound desc = could not find container \"3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120\": container with ID starting with 3d5f3c5da6fae6b9a892c5ea565fa4f931ffe74f53b1e679d490d2f1d07df120 not found: ID does not exist" Oct 30 18:43:44.996641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.996537 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shg55\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-kube-api-access-shg55\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:44.996641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:44.996569 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3ea0d22-b1cd-4eca-916b-c5155ab1139c-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:43:45.026870 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:45.026849 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t"] Oct 30 18:43:45.032759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:45.032739 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-657cbnz8t"] Oct 30 18:43:45.071233 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:45.063118 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ea0d22-b1cd-4eca-916b-c5155ab1139c" path="/var/lib/kubelet/pods/d3ea0d22-b1cd-4eca-916b-c5155ab1139c/volumes" Oct 30 18:43:48.897411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:48.897346 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:43:49.872487 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:43:49.872447 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:43:50.350109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.350073 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs"] Oct 30 18:43:50.350670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.350655 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d3ea0d22-b1cd-4eca-916b-c5155ab1139c" containerName="payload-processing" Oct 30 18:43:50.350729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.350672 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ea0d22-b1cd-4eca-916b-c5155ab1139c" containerName="payload-processing" Oct 30 18:43:50.350817 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.350807 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ea0d22-b1cd-4eca-916b-c5155ab1139c" containerName="payload-processing" Oct 30 18:43:50.354061 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.354043 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.373038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.373014 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"agc-cred-12bcdc53060d\"" Oct 30 18:43:50.373169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.373123 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"payload-processing-external-model-e2e--40d660e1-dockercfg-czmv5\"" Oct 30 18:43:50.385344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.385320 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-713dc8\"" Oct 30 18:43:50.385604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.385522 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"openshift-service-ca.crt\"" Oct 30 18:43:50.385712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.385569 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"routing-overlay\"" Oct 30 18:43:50.385712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.385571 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"kube-root-ca.crt\"" Oct 30 18:43:50.390633 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.390607 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs"] Oct 30 18:43:50.460633 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.460506 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.461373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.461219 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgd59\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-kube-api-access-sgd59\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.461746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.461726 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.462202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.462160 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.568288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.565603 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.568288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.565675 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.568288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.565920 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.568288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.565955 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgd59\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-kube-api-access-sgd59\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.568288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.567880 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.568288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.568138 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.573287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.571926 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.590380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.587236 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgd59\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-kube-api-access-sgd59\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:50.668882 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:50.668806 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:51.000996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:51.000971 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs"] Oct 30 18:43:51.001570 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:43:51.001543 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a8a996_6dbd_4a2e_a63c_88421176c7f9.slice/crio-42501e9996c38dff4b04a5919ffd540b5cb951d31ff9c7e083dcfedbede77b90 WatchSource:0}: Error finding container 42501e9996c38dff4b04a5919ffd540b5cb951d31ff9c7e083dcfedbede77b90: Status 404 returned error can't find the container with id 42501e9996c38dff4b04a5919ffd540b5cb951d31ff9c7e083dcfedbede77b90 Oct 30 18:43:51.044932 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:51.042197 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" event={"ID":"06a8a996-6dbd-4a2e-a63c-88421176c7f9","Type":"ContainerStarted","Data":"42501e9996c38dff4b04a5919ffd540b5cb951d31ff9c7e083dcfedbede77b90"} Oct 30 18:43:52.050480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:52.050441 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" event={"ID":"06a8a996-6dbd-4a2e-a63c-88421176c7f9","Type":"ContainerStarted","Data":"ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492"} Oct 30 18:43:52.050964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:52.050527 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:43:52.090439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:52.090396 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" podStartSLOduration=2.09038184 podStartE2EDuration="2.09038184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:52.089170243 +0000 UTC m=+2731.138215147" watchObservedRunningTime="2026-10-30 18:43:52.09038184 +0000 UTC m=+2731.139426753" Oct 30 18:43:53.783652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.783616 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5"] Oct 30 18:43:53.787933 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.787913 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:53.807403 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.807378 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5"] Oct 30 18:43:53.816657 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.816627 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"agc-cred-90bdaaae67f6\"" Oct 30 18:43:53.816657 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.816645 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-e2e--6d8a874f-dockercfg-rwjcz\"" Oct 30 18:43:53.816868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.816667 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6018b4\"" Oct 30 18:43:53.816868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.816722 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"routing-overlay\"" Oct 30 18:43:53.816868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.816737 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"openshift-service-ca.crt\"" Oct 30 18:43:53.819765 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.819749 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"kube-root-ca.crt\"" Oct 30 18:43:53.920178 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.920136 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czl4z\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-kube-api-access-czl4z\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:53.920670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.920630 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:53.920909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.920878 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:53.921049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:53.921034 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.022700 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.022664 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czl4z\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-kube-api-access-czl4z\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.022912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.022740 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.022912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.022769 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.022912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.022806 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.023451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.023427 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.023575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.023558 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.025554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.025534 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.042377 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.042330 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czl4z\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-kube-api-access-czl4z\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.102276 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.102233 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:54.411024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:54.410997 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5"] Oct 30 18:43:54.411907 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:43:54.411881 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf5b6b7_eb3c_49ae_9cb5_ae7ec8b76f28.slice/crio-823174dcb8a90ff765f297534f79636d1de4dcc86a11ff4edf1a2b54dcf0c54a WatchSource:0}: Error finding container 823174dcb8a90ff765f297534f79636d1de4dcc86a11ff4edf1a2b54dcf0c54a: Status 404 returned error can't find the container with id 823174dcb8a90ff765f297534f79636d1de4dcc86a11ff4edf1a2b54dcf0c54a Oct 30 18:43:55.079209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:55.079173 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" event={"ID":"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28","Type":"ContainerStarted","Data":"b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005"} Oct 30 18:43:55.079669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:55.079227 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" event={"ID":"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28","Type":"ContainerStarted","Data":"823174dcb8a90ff765f297534f79636d1de4dcc86a11ff4edf1a2b54dcf0c54a"} Oct 30 18:43:55.079669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:55.079291 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:43:55.120825 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:43:55.120780 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" podStartSLOduration=2.12076484 podStartE2EDuration="2.12076484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:55.117529496 +0000 UTC m=+2734.166574410" watchObservedRunningTime="2026-10-30 18:43:55.12076484 +0000 UTC m=+2734.169809754" Oct 30 18:44:03.063034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:03.062961 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:44:06.089858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:06.089829 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:44:19.872467 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:44:19.872427 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:44:21.936888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:21.936852 4862 scope.go:122] "RemoveContainer" containerID="963da578a15a1a54e3dfa1c265f20b201ffec69281578db9647a8cfcac6c1370" Oct 30 18:44:26.745279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:26.745241 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf"] Oct 30 18:44:26.750903 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:26.750887 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:26.768499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:26.768478 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf"] Oct 30 18:44:26.777523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:26.777505 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6babca\"/\"agc-cred-47f047052fe6\"" Oct 30 18:44:26.927794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:26.927769 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:26.927916 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:26.927803 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:26.927916 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:26.927838 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:26.927916 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:26.927905 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxjnv\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-kube-api-access-bxjnv\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.028780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.028721 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.028780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.028757 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.028780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.028779 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.029003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.028799 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxjnv\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-kube-api-access-bxjnv\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.029396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.029374 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.029671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.029465 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.031565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.031536 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.048251 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.048223 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxjnv\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-kube-api-access-bxjnv\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.069393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.069362 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:27.369376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:27.369336 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf"] Oct 30 18:44:27.371061 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:44:27.371032 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9870774_1319_40f7_92d9_74138973540f.slice/crio-68a987ebcbba798c1bb0cea6df5504087dfc64ee4cbb552ceff92b9b0d429278 WatchSource:0}: Error finding container 68a987ebcbba798c1bb0cea6df5504087dfc64ee4cbb552ceff92b9b0d429278: Status 404 returned error can't find the container with id 68a987ebcbba798c1bb0cea6df5504087dfc64ee4cbb552ceff92b9b0d429278 Oct 30 18:44:28.297161 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:28.297126 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" event={"ID":"f9870774-1319-40f7-92d9-74138973540f","Type":"ContainerStarted","Data":"43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd"} Oct 30 18:44:28.297563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:28.297184 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" event={"ID":"f9870774-1319-40f7-92d9-74138973540f","Type":"ContainerStarted","Data":"68a987ebcbba798c1bb0cea6df5504087dfc64ee4cbb552ceff92b9b0d429278"} Oct 30 18:44:28.297563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:28.297344 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:28.338111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:28.338072 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" podStartSLOduration=2.338059754 podStartE2EDuration="2.338059754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:28.335935024 +0000 UTC m=+2767.384979940" watchObservedRunningTime="2026-10-30 18:44:28.338059754 +0000 UTC m=+2767.387104667" Oct 30 18:44:39.307693 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:39.307668 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:44:39.380557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:39.380527 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7"] Oct 30 18:44:39.570181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:39.570097 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:44:44.247186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.247154 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4"] Oct 30 18:44:44.252579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.252553 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.270946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.270921 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4"] Oct 30 18:44:44.279254 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.279235 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:44:44.279374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.279293 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:44:44.328733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.328703 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a96e4e-48jc4\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.328859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.328747 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8cqc\" (UniqueName: \"kubernetes.io/projected/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-kube-api-access-c8cqc\") pod \"maas-api-revoke-sub-94a96e4e-48jc4\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.328992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.328972 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a96e4e-48jc4\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.381218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.381184 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" podUID="13b0150b-579c-4ea4-abf8-506418c68165" containerName="payload-processing" containerID="cri-o://ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5" gracePeriod=25 Oct 30 18:44:44.430332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.430299 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a96e4e-48jc4\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.430457 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.430355 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8cqc\" (UniqueName: \"kubernetes.io/projected/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-kube-api-access-c8cqc\") pod \"maas-api-revoke-sub-94a96e4e-48jc4\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.430524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.430508 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a96e4e-48jc4\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.431124 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.431102 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94a96e4e-48jc4\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.432731 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.432710 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94a96e4e-48jc4\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.450397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.450374 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8cqc\" (UniqueName: \"kubernetes.io/projected/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-kube-api-access-c8cqc\") pod \"maas-api-revoke-sub-94a96e4e-48jc4\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.566600 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.566368 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:44.656751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.656727 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:44:44.734992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.734962 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-provider-credentials\") pod \"13b0150b-579c-4ea4-abf8-506418c68165\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " Oct 30 18:44:44.735161 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.735078 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-plugins-config-volume\") pod \"13b0150b-579c-4ea4-abf8-506418c68165\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " Oct 30 18:44:44.735161 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.735136 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-routing-overlay-volume\") pod \"13b0150b-579c-4ea4-abf8-506418c68165\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " Oct 30 18:44:44.735319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.735230 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-kube-api-access-hjtm5\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-kube-api-access-hjtm5\") pod \"13b0150b-579c-4ea4-abf8-506418c68165\" (UID: \"13b0150b-579c-4ea4-abf8-506418c68165\") " Oct 30 18:44:44.735572 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.735527 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-plugins-config-volume" pod "13b0150b-579c-4ea4-abf8-506418c68165" (UID: "13b0150b-579c-4ea4-abf8-506418c68165"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:44.735640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.735580 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-routing-overlay-volume" pod "13b0150b-579c-4ea4-abf8-506418c68165" (UID: "13b0150b-579c-4ea4-abf8-506418c68165"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:44.735813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.735789 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:44:44.735813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.735814 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13b0150b-579c-4ea4-abf8-506418c68165-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:44:44.737422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.737401 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-provider-credentials" pod "13b0150b-579c-4ea4-abf8-506418c68165" (UID: "13b0150b-579c-4ea4-abf8-506418c68165"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:44.737519 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.737445 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-kube-api-access-hjtm5" pod "13b0150b-579c-4ea4-abf8-506418c68165" (UID: "13b0150b-579c-4ea4-abf8-506418c68165"). InnerVolumeSpecName "kube-api-access-hjtm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:44.837523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.837451 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjtm5\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-kube-api-access-hjtm5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:44:44.837523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.837477 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13b0150b-579c-4ea4-abf8-506418c68165-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:44:44.883354 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:44.883327 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4"] Oct 30 18:44:44.883774 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:44:44.883755 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3398ceb5_ee5d_4d8e_b7cf_0c7af5cf44ea.slice/crio-c6b832f632f60937875f48796c2e7680d608922481d5e7d2d056bd9c9609ae42 WatchSource:0}: Error finding container c6b832f632f60937875f48796c2e7680d608922481d5e7d2d056bd9c9609ae42: Status 404 returned error can't find the container with id c6b832f632f60937875f48796c2e7680d608922481d5e7d2d056bd9c9609ae42 Oct 30 18:44:45.412134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.411862 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" event={"ID":"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea","Type":"ContainerStarted","Data":"431bc637781670ea98ce934bc6eacd4314ca60b7b69f866879093a8220f2ee80"} Oct 30 18:44:45.412134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.411921 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" event={"ID":"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea","Type":"ContainerStarted","Data":"c6b832f632f60937875f48796c2e7680d608922481d5e7d2d056bd9c9609ae42"} Oct 30 18:44:45.413944 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.413921 4862 generic.go:356] "Generic (PLEG): container finished" podID="13b0150b-579c-4ea4-abf8-506418c68165" containerID="ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5" exitCode=0 Oct 30 18:44:45.414077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.413970 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" Oct 30 18:44:45.414077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.414002 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" event={"ID":"13b0150b-579c-4ea4-abf8-506418c68165","Type":"ContainerDied","Data":"ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5"} Oct 30 18:44:45.414077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.414060 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7" event={"ID":"13b0150b-579c-4ea4-abf8-506418c68165","Type":"ContainerDied","Data":"bb70ef547b480796e708d79bb42fb95351c95a49a14962b6d0816e63c799190b"} Oct 30 18:44:45.414246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.414090 4862 scope.go:122] "RemoveContainer" containerID="ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5" Oct 30 18:44:45.424568 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.424545 4862 scope.go:122] "RemoveContainer" containerID="ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5" Oct 30 18:44:45.424846 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:44:45.424826 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5\": container with ID starting with ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5 not found: ID does not exist" containerID="ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5" Oct 30 18:44:45.424925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.424853 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5"} err="failed to get container status \"ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5\": rpc error: code = NotFound desc = could not find container \"ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5\": container with ID starting with ca8cb2b4b155413b383088b79cc3a40b3c687c63d0976ace19e2c9ec6ca09eb5 not found: ID does not exist" Oct 30 18:44:45.451849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.451812 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" podStartSLOduration=1.451800506 podStartE2EDuration="1.451800506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:45.450511406 +0000 UTC m=+2784.499556318" watchObservedRunningTime="2026-10-30 18:44:45.451800506 +0000 UTC m=+2784.500845418" Oct 30 18:44:45.486355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.486332 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7"] Oct 30 18:44:45.489024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:45.489005 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769b964n7"] Oct 30 18:44:46.421868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:46.421834 4862 generic.go:356] "Generic (PLEG): container finished" podID="3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea" containerID="431bc637781670ea98ce934bc6eacd4314ca60b7b69f866879093a8220f2ee80" exitCode=0 Oct 30 18:44:46.422292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:46.421905 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" event={"ID":"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea","Type":"ContainerDied","Data":"431bc637781670ea98ce934bc6eacd4314ca60b7b69f866879093a8220f2ee80"} Oct 30 18:44:47.062780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.062752 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b0150b-579c-4ea4-abf8-506418c68165" path="/var/lib/kubelet/pods/13b0150b-579c-4ea4-abf8-506418c68165/volumes" Oct 30 18:44:47.596425 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.596402 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:47.669630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.669602 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-openshift-ca\") pod \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " Oct 30 18:44:47.669810 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.669645 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-xks-ca\") pod \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " Oct 30 18:44:47.669810 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.669702 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-kube-api-access-c8cqc\" (UniqueName: \"kubernetes.io/projected/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-kube-api-access-c8cqc\") pod \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\" (UID: \"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea\") " Oct 30 18:44:47.670021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.669998 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-openshift-ca" pod "3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea" (UID: "3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:47.670125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.670101 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:44:47.671974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.671953 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-xks-ca" pod "3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea" (UID: "3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:47.672127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.672104 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-kube-api-access-c8cqc" pod "3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea" (UID: "3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea"). InnerVolumeSpecName "kube-api-access-c8cqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:47.771466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.771445 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:44:47.771466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:47.771466 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8cqc\" (UniqueName: \"kubernetes.io/projected/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea-kube-api-access-c8cqc\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:44:48.435756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:48.435725 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" Oct 30 18:44:48.435918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:48.435760 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4" event={"ID":"3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea","Type":"ContainerDied","Data":"c6b832f632f60937875f48796c2e7680d608922481d5e7d2d056bd9c9609ae42"} Oct 30 18:44:48.435918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:48.435809 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b832f632f60937875f48796c2e7680d608922481d5e7d2d056bd9c9609ae42" Oct 30 18:44:49.869280 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:44:49.869230 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:44:55.435184 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:55.435151 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2"] Oct 30 18:44:59.219014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.218981 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh"] Oct 30 18:44:59.219629 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.219614 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea" containerName="revoke-keys" Oct 30 18:44:59.219685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.219631 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea" containerName="revoke-keys" Oct 30 18:44:59.219685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.219642 4862 cpu_manager.go:414] "containerMap: removing container" podUID="13b0150b-579c-4ea4-abf8-506418c68165" containerName="payload-processing" Oct 30 18:44:59.219685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.219649 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b0150b-579c-4ea4-abf8-506418c68165" containerName="payload-processing" Oct 30 18:44:59.219823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.219738 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea" containerName="revoke-keys" Oct 30 18:44:59.219823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.219748 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b0150b-579c-4ea4-abf8-506418c68165" containerName="payload-processing" Oct 30 18:44:59.223163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.223147 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.246303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.246276 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh"] Oct 30 18:44:59.255341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.255313 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"agc-cred-ccddb066b30b\"" Oct 30 18:44:59.255460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.255373 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"payload-processing-external-model-e2e--41b9a229-dockercfg-7w9tm\"" Oct 30 18:44:59.332285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.332156 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.332531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.332306 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fd5k\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-kube-api-access-9fd5k\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.332531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.332404 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.332531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.332432 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.433984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.433954 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.434156 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.434065 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fd5k\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-kube-api-access-9fd5k\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.434206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.434164 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.434206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.434195 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.434805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.434780 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.434805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.434805 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.436730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.436711 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.453773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.453751 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fd5k\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-kube-api-access-9fd5k\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.536842 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.536778 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:44:59.848086 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:44:59.848059 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh"] Oct 30 18:44:59.849178 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:44:59.849153 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04094c2_6fe0_4ae1_a731_b8679713e79f.slice/crio-2341c4920735648eb4a1491871ba038d61ddb49d49f228564736b88a32dd4aa3 WatchSource:0}: Error finding container 2341c4920735648eb4a1491871ba038d61ddb49d49f228564736b88a32dd4aa3: Status 404 returned error can't find the container with id 2341c4920735648eb4a1491871ba038d61ddb49d49f228564736b88a32dd4aa3 Oct 30 18:45:00.151507 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.151426 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24"] Oct 30 18:45:00.155619 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.155595 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm"] Oct 30 18:45:00.155786 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.155703 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:00.159478 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.159460 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg"] Oct 30 18:45:00.159590 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.159575 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:00.163197 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.163179 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:00.174582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.174555 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z"] Oct 30 18:45:00.179153 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.179138 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:00.180413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.180397 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:45:00.185302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.185257 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm"] Oct 30 18:45:00.189233 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.189215 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg"] Oct 30 18:45:00.198393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.198370 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg"] Oct 30 18:45:00.198512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.198493 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:00.204450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.204428 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww"] Oct 30 18:45:00.209866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.209848 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:00.210772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.210750 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24"] Oct 30 18:45:00.228758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.228734 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg"] Oct 30 18:45:00.237681 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.237657 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z"] Oct 30 18:45:00.246030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.246011 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww"] Oct 30 18:45:00.249006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.248989 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc"] Oct 30 18:45:00.253130 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.253113 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk"] Oct 30 18:45:00.253249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.253232 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:00.259771 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.258033 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:00.265941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.265878 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc"] Oct 30 18:45:00.269277 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.269240 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk"] Oct 30 18:45:00.273128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.273104 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv"] Oct 30 18:45:00.279556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.279537 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m"] Oct 30 18:45:00.282186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.282167 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:00.285906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.285886 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv"] Oct 30 18:45:00.286026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.286008 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:00.291936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.291872 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:00.310283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.306795 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv"] Oct 30 18:45:00.313188 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.312713 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m"] Oct 30 18:45:00.315376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.315351 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv"] Oct 30 18:45:00.317289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.317245 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn"] Oct 30 18:45:00.322526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.322508 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf"] Oct 30 18:45:00.322697 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.322673 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.349918 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfffb210-225d-4707-9bdd-4608be2dc749-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm\" (UID: \"cfffb210-225d-4707-9bdd-4608be2dc749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.349976 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k75g\" (UniqueName: \"kubernetes.io/projected/cfffb210-225d-4707-9bdd-4608be2dc749-kube-api-access-8k75g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm\" (UID: \"cfffb210-225d-4707-9bdd-4608be2dc749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350114 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2v5k\" (UniqueName: \"kubernetes.io/projected/dc189cb4-8886-479c-903a-752c5ad19901-kube-api-access-m2v5k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg\" (UID: \"dc189cb4-8886-479c-903a-752c5ad19901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350154 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24\" (UID: \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350189 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cf935f-86dc-4790-93b2-c7baea9b6a42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww\" (UID: \"06cf935f-86dc-4790-93b2-c7baea9b6a42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350216 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc189cb4-8886-479c-903a-752c5ad19901-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg\" (UID: \"dc189cb4-8886-479c-903a-752c5ad19901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350302 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclrt\" (UniqueName: \"kubernetes.io/projected/1ac031fb-b03c-4a94-9623-e19a1f4296e5-kube-api-access-cclrt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg\" (UID: \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350347 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544dr\" (UniqueName: \"kubernetes.io/projected/edb49158-4950-4fcb-9ddc-aaa5005a019f-kube-api-access-544dr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z\" (UID: \"edb49158-4950-4fcb-9ddc-aaa5005a019f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350390 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc\" (UID: \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350452 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcjpt\" (UniqueName: \"kubernetes.io/projected/06cf935f-86dc-4790-93b2-c7baea9b6a42-kube-api-access-dcjpt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww\" (UID: \"06cf935f-86dc-4790-93b2-c7baea9b6a42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350548 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkv7j\" (UniqueName: \"kubernetes.io/projected/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-kube-api-access-qkv7j\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24\" (UID: \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350656 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb49158-4950-4fcb-9ddc-aaa5005a019f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z\" (UID: \"edb49158-4950-4fcb-9ddc-aaa5005a019f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350731 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac031fb-b03c-4a94-9623-e19a1f4296e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg\" (UID: \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.350849 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc766\" (UniqueName: \"kubernetes.io/projected/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-kube-api-access-pc766\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc\" (UID: \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:00.364482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.355903 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:00.391287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.377329 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn"] Oct 30 18:45:00.391287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.380233 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf"] Oct 30 18:45:00.438441 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.437157 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" podUID="288f283c-eff8-4ade-9b4b-6543280108a3" containerName="payload-processing" containerID="cri-o://f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8" gracePeriod=25 Oct 30 18:45:00.453000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.452969 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72wfj\" (UniqueName: \"kubernetes.io/projected/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-kube-api-access-72wfj\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv\" (UID: \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:00.453148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453021 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2v5k\" (UniqueName: \"kubernetes.io/projected/dc189cb4-8886-479c-903a-752c5ad19901-kube-api-access-m2v5k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg\" (UID: \"dc189cb4-8886-479c-903a-752c5ad19901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:00.453148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453093 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24\" (UID: \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:00.453148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453131 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cf935f-86dc-4790-93b2-c7baea9b6a42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww\" (UID: \"06cf935f-86dc-4790-93b2-c7baea9b6a42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:00.453148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453149 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc189cb4-8886-479c-903a-752c5ad19901-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg\" (UID: \"dc189cb4-8886-479c-903a-752c5ad19901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:00.453390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453208 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclrt\" (UniqueName: \"kubernetes.io/projected/1ac031fb-b03c-4a94-9623-e19a1f4296e5-kube-api-access-cclrt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg\" (UID: \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:00.453390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453248 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6m9\" (UniqueName: \"kubernetes.io/projected/bb2fed97-f53d-4f11-a967-8b975837a13e-kube-api-access-cg6m9\") pod \"maas-api-key-cleanup-29889765-dtlxv\" (UID: \"bb2fed97-f53d-4f11-a967-8b975837a13e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:00.453390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453296 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544dr\" (UniqueName: \"kubernetes.io/projected/edb49158-4950-4fcb-9ddc-aaa5005a019f-kube-api-access-544dr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z\" (UID: \"edb49158-4950-4fcb-9ddc-aaa5005a019f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:00.453390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453322 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv\" (UID: \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:00.453390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453368 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc\" (UID: \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453443 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjpt\" (UniqueName: \"kubernetes.io/projected/06cf935f-86dc-4790-93b2-c7baea9b6a42-kube-api-access-dcjpt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww\" (UID: \"06cf935f-86dc-4790-93b2-c7baea9b6a42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453485 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkv7j\" (UniqueName: \"kubernetes.io/projected/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-kube-api-access-qkv7j\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24\" (UID: \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453528 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2168db-5894-4b73-ba78-754f1d8863bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf\" (UID: \"bb2168db-5894-4b73-ba78-754f1d8863bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453553 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2fed97-f53d-4f11-a967-8b975837a13e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-dtlxv\" (UID: \"bb2fed97-f53d-4f11-a967-8b975837a13e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453588 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn\" (UID: \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453771 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb49158-4950-4fcb-9ddc-aaa5005a019f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z\" (UID: \"edb49158-4950-4fcb-9ddc-aaa5005a019f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453891 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac031fb-b03c-4a94-9623-e19a1f4296e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg\" (UID: \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.453935 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9mz7\" (UniqueName: \"kubernetes.io/projected/dbf488c3-95ac-4149-9b8b-ebb4522cc909-kube-api-access-k9mz7\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk\" (UID: \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.454008 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djpfd\" (UniqueName: \"kubernetes.io/projected/8c1bb6bd-ca03-45c2-b563-0f884658acd0-kube-api-access-djpfd\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m\" (UID: \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.454042 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf488c3-95ac-4149-9b8b-ebb4522cc909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk\" (UID: \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.454109 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc766\" (UniqueName: \"kubernetes.io/projected/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-kube-api-access-pc766\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc\" (UID: \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.454152 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68kt\" (UniqueName: \"kubernetes.io/projected/bb2168db-5894-4b73-ba78-754f1d8863bc-kube-api-access-x68kt\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf\" (UID: \"bb2168db-5894-4b73-ba78-754f1d8863bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.454178 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bb6bd-ca03-45c2-b563-0f884658acd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m\" (UID: \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.454177 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24\" (UID: \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.454557 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc189cb4-8886-479c-903a-752c5ad19901-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg\" (UID: \"dc189cb4-8886-479c-903a-752c5ad19901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.454673 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfffb210-225d-4707-9bdd-4608be2dc749-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm\" (UID: \"cfffb210-225d-4707-9bdd-4608be2dc749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.454767 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cf935f-86dc-4790-93b2-c7baea9b6a42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww\" (UID: \"06cf935f-86dc-4790-93b2-c7baea9b6a42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.455719 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb49158-4950-4fcb-9ddc-aaa5005a019f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z\" (UID: \"edb49158-4950-4fcb-9ddc-aaa5005a019f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.455967 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k75g\" (UniqueName: \"kubernetes.io/projected/cfffb210-225d-4707-9bdd-4608be2dc749-kube-api-access-8k75g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm\" (UID: \"cfffb210-225d-4707-9bdd-4608be2dc749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.456049 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac031fb-b03c-4a94-9623-e19a1f4296e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg\" (UID: \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.456192 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sw4x\" (UniqueName: \"kubernetes.io/projected/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-kube-api-access-4sw4x\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn\" (UID: \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.456367 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc\" (UID: \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:00.457084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.456821 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfffb210-225d-4707-9bdd-4608be2dc749-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm\" (UID: \"cfffb210-225d-4707-9bdd-4608be2dc749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:00.474095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.474066 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2v5k\" (UniqueName: \"kubernetes.io/projected/dc189cb4-8886-479c-903a-752c5ad19901-kube-api-access-m2v5k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg\" (UID: \"dc189cb4-8886-479c-903a-752c5ad19901\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:00.475169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.475146 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclrt\" (UniqueName: \"kubernetes.io/projected/1ac031fb-b03c-4a94-9623-e19a1f4296e5-kube-api-access-cclrt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg\" (UID: \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:00.477194 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.477035 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-544dr\" (UniqueName: \"kubernetes.io/projected/edb49158-4950-4fcb-9ddc-aaa5005a019f-kube-api-access-544dr\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z\" (UID: \"edb49158-4950-4fcb-9ddc-aaa5005a019f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:00.477194 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.477044 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k75g\" (UniqueName: \"kubernetes.io/projected/cfffb210-225d-4707-9bdd-4608be2dc749-kube-api-access-8k75g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm\" (UID: \"cfffb210-225d-4707-9bdd-4608be2dc749\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:00.478681 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.478663 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:00.478996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.478974 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjpt\" (UniqueName: \"kubernetes.io/projected/06cf935f-86dc-4790-93b2-c7baea9b6a42-kube-api-access-dcjpt\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww\" (UID: \"06cf935f-86dc-4790-93b2-c7baea9b6a42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:00.480961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.480942 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc766\" (UniqueName: \"kubernetes.io/projected/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-kube-api-access-pc766\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc\" (UID: \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:00.481451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.481434 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkv7j\" (UniqueName: \"kubernetes.io/projected/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-kube-api-access-qkv7j\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24\" (UID: \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:00.487331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.487302 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:00.514888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.514661 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:00.575234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.574975 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.578240 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9mz7\" (UniqueName: \"kubernetes.io/projected/dbf488c3-95ac-4149-9b8b-ebb4522cc909-kube-api-access-k9mz7\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk\" (UID: \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.578525 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djpfd\" (UniqueName: \"kubernetes.io/projected/8c1bb6bd-ca03-45c2-b563-0f884658acd0-kube-api-access-djpfd\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m\" (UID: \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.578556 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf488c3-95ac-4149-9b8b-ebb4522cc909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk\" (UID: \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.578629 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68kt\" (UniqueName: \"kubernetes.io/projected/bb2168db-5894-4b73-ba78-754f1d8863bc-kube-api-access-x68kt\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf\" (UID: \"bb2168db-5894-4b73-ba78-754f1d8863bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.578661 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bb6bd-ca03-45c2-b563-0f884658acd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m\" (UID: \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.578729 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sw4x\" (UniqueName: \"kubernetes.io/projected/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-kube-api-access-4sw4x\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn\" (UID: \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.578758 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72wfj\" (UniqueName: \"kubernetes.io/projected/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-kube-api-access-72wfj\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv\" (UID: \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.578857 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6m9\" (UniqueName: \"kubernetes.io/projected/bb2fed97-f53d-4f11-a967-8b975837a13e-kube-api-access-cg6m9\") pod \"maas-api-key-cleanup-29889765-dtlxv\" (UID: \"bb2fed97-f53d-4f11-a967-8b975837a13e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.578903 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv\" (UID: \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.579012 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2168db-5894-4b73-ba78-754f1d8863bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf\" (UID: \"bb2168db-5894-4b73-ba78-754f1d8863bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.579037 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2fed97-f53d-4f11-a967-8b975837a13e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-dtlxv\" (UID: \"bb2fed97-f53d-4f11-a967-8b975837a13e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.579067 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn\" (UID: \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.579670 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.579880 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn\" (UID: \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.580816 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv\" (UID: \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.581484 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2168db-5894-4b73-ba78-754f1d8863bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf\" (UID: \"bb2168db-5894-4b73-ba78-754f1d8863bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.582090 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2fed97-f53d-4f11-a967-8b975837a13e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-dtlxv\" (UID: \"bb2fed97-f53d-4f11-a967-8b975837a13e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.582730 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf488c3-95ac-4149-9b8b-ebb4522cc909-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk\" (UID: \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.583683 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bb6bd-ca03-45c2-b563-0f884658acd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m\" (UID: \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:00.589989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.583796 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:00.824338 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.824282 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6m9\" (UniqueName: \"kubernetes.io/projected/bb2fed97-f53d-4f11-a967-8b975837a13e-kube-api-access-cg6m9\") pod \"maas-api-key-cleanup-29889765-dtlxv\" (UID: \"bb2fed97-f53d-4f11-a967-8b975837a13e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:00.831041 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.830843 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sw4x\" (UniqueName: \"kubernetes.io/projected/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-kube-api-access-4sw4x\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn\" (UID: \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:00.843175 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.831623 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:00.843175 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.840976 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9mz7\" (UniqueName: \"kubernetes.io/projected/dbf488c3-95ac-4149-9b8b-ebb4522cc909-kube-api-access-k9mz7\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk\" (UID: \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:00.849527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.849497 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68kt\" (UniqueName: \"kubernetes.io/projected/bb2168db-5894-4b73-ba78-754f1d8863bc-kube-api-access-x68kt\") pod \"maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf\" (UID: \"bb2168db-5894-4b73-ba78-754f1d8863bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:00.849992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.849967 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:00.851485 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.851462 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djpfd\" (UniqueName: \"kubernetes.io/projected/8c1bb6bd-ca03-45c2-b563-0f884658acd0-kube-api-access-djpfd\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m\" (UID: \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:00.856350 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.855740 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" event={"ID":"d04094c2-6fe0-4ae1-a731-b8679713e79f","Type":"ContainerStarted","Data":"0dec8ee1a98d1008c3e60100da21df00224bcf304d3385f0099df24502954ab5"} Oct 30 18:45:00.856350 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.855834 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" event={"ID":"d04094c2-6fe0-4ae1-a731-b8679713e79f","Type":"ContainerStarted","Data":"2341c4920735648eb4a1491871ba038d61ddb49d49f228564736b88a32dd4aa3"} Oct 30 18:45:00.857108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.856868 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:45:00.860226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.859892 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72wfj\" (UniqueName: \"kubernetes.io/projected/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-kube-api-access-72wfj\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv\" (UID: \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:00.895358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.891347 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx"] Oct 30 18:45:00.895704 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.895534 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:00.943359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.926857 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx"] Oct 30 18:45:00.943359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.931487 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:00.945074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.944426 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:00.945074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:00.944986 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:00.977604 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:00.963795 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:00.977604 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:00.972886 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:00.977604 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:00.975339 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials podName:d04094c2-6fe0-4ae1-a731-b8679713e79f nodeName:}" failed. No retries permitted until 2026-10-30 18:45:01.474476146 +0000 UTC m=+2800.523521038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" (UID: "d04094c2-6fe0-4ae1-a731-b8679713e79f") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.013519 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.013304 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:01.018772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.018680 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"e2e-cfl-3695cd-provider-cred\"" Oct 30 18:45:01.053613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.053575 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.053718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.053660 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f7d\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-kube-api-access-w5f7d\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.067623 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.053747 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.067623 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.053948 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.067623 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.066326 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:01.164298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.155960 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.164298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.156161 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.164298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.156226 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f7d\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-kube-api-access-w5f7d\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.164298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.156334 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.164298 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.159974 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.164298 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.160110 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.164298 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.160304 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials podName:f5920a67-84ce-4ede-896e-2de205b0f63d nodeName:}" failed. No retries permitted until 2026-10-30 18:45:01.660155158 +0000 UTC m=+2800.709200049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" (UID: "f5920a67-84ce-4ede-896e-2de205b0f63d") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.164298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.161045 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.164298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.163280 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.180784 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.180170 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm"] Oct 30 18:45:01.212408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.211713 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:45:01.245159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.245101 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" podStartSLOduration=2.245074556 podStartE2EDuration="2.245074556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:01.236785028 +0000 UTC m=+2800.285829954" watchObservedRunningTime="2026-10-30 18:45:01.245074556 +0000 UTC m=+2800.294119470" Oct 30 18:45:01.248380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.248319 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f7d\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-kube-api-access-w5f7d\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.359223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.359193 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-kube-api-access-sxqzq\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-kube-api-access-sxqzq\") pod \"288f283c-eff8-4ade-9b4b-6543280108a3\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " Oct 30 18:45:01.359393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.359241 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-provider-credentials\") pod \"288f283c-eff8-4ade-9b4b-6543280108a3\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " Oct 30 18:45:01.359393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.359287 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-plugins-config-volume\") pod \"288f283c-eff8-4ade-9b4b-6543280108a3\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " Oct 30 18:45:01.359393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.359323 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-routing-overlay-volume\") pod \"288f283c-eff8-4ade-9b4b-6543280108a3\" (UID: \"288f283c-eff8-4ade-9b4b-6543280108a3\") " Oct 30 18:45:01.359809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.359781 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-routing-overlay-volume" pod "288f283c-eff8-4ade-9b4b-6543280108a3" (UID: "288f283c-eff8-4ade-9b4b-6543280108a3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:01.359809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.359796 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-plugins-config-volume" pod "288f283c-eff8-4ade-9b4b-6543280108a3" (UID: "288f283c-eff8-4ade-9b4b-6543280108a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:01.360089 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.360069 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:01.360089 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.360091 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/288f283c-eff8-4ade-9b4b-6543280108a3-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:01.361764 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.361745 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-kube-api-access-sxqzq" pod "288f283c-eff8-4ade-9b4b-6543280108a3" (UID: "288f283c-eff8-4ade-9b4b-6543280108a3"). InnerVolumeSpecName "kube-api-access-sxqzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:01.361845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.361775 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-provider-credentials" pod "288f283c-eff8-4ade-9b4b-6543280108a3" (UID: "288f283c-eff8-4ade-9b4b-6543280108a3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:01.461560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.461526 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxqzq\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-kube-api-access-sxqzq\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:01.461560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.461561 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/288f283c-eff8-4ade-9b4b-6543280108a3-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:01.475929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.475902 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww"] Oct 30 18:45:01.478812 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:01.478780 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cf935f_86dc_4790_93b2_c7baea9b6a42.slice/crio-d86b2a47ad26d6522adaaf0033c5ad076fa965e50695d07941117542e75c7bcc WatchSource:0}: Error finding container d86b2a47ad26d6522adaaf0033c5ad076fa965e50695d07941117542e75c7bcc: Status 404 returned error can't find the container with id d86b2a47ad26d6522adaaf0033c5ad076fa965e50695d07941117542e75c7bcc Oct 30 18:45:01.490143 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.490114 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z"] Oct 30 18:45:01.494501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.494475 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc"] Oct 30 18:45:01.564448 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.563154 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.564448 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.563179 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.564448 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.563232 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials podName:d04094c2-6fe0-4ae1-a731-b8679713e79f nodeName:}" failed. No retries permitted until 2026-10-30 18:45:02.563215342 +0000 UTC m=+2801.612260249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" (UID: "d04094c2-6fe0-4ae1-a731-b8679713e79f") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.564946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.564898 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg"] Oct 30 18:45:01.572509 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:01.567056 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac031fb_b03c_4a94_9623_e19a1f4296e5.slice/crio-5f19654f06a658e6e9c50925eeba07fa8d2c33465f1ae03265a9c5bb474eacaa WatchSource:0}: Error finding container 5f19654f06a658e6e9c50925eeba07fa8d2c33465f1ae03265a9c5bb474eacaa: Status 404 returned error can't find the container with id 5f19654f06a658e6e9c50925eeba07fa8d2c33465f1ae03265a9c5bb474eacaa Oct 30 18:45:01.591218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.591194 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg"] Oct 30 18:45:01.592453 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:01.592427 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef WatchSource:0}: Error finding container d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef: Status 404 returned error can't find the container with id d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef Oct 30 18:45:01.599294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.599244 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24"] Oct 30 18:45:01.601378 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:01.601353 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97dc6f5_10b4_4d60_90bf_423a8da7ab74.slice/crio-15c286bfd3d4ef561ac5e1188805d15fa28e5f844beb702f44fa8d05f5748ad9 WatchSource:0}: Error finding container 15c286bfd3d4ef561ac5e1188805d15fa28e5f844beb702f44fa8d05f5748ad9: Status 404 returned error can't find the container with id 15c286bfd3d4ef561ac5e1188805d15fa28e5f844beb702f44fa8d05f5748ad9 Oct 30 18:45:01.651094 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.651068 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh"] Oct 30 18:45:01.664059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.664032 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:01.664238 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.664223 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.664318 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.664245 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.664367 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.664316 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials podName:f5920a67-84ce-4ede-896e-2de205b0f63d nodeName:}" failed. No retries permitted until 2026-10-30 18:45:02.664296977 +0000 UTC m=+2801.713341888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" (UID: "f5920a67-84ce-4ede-896e-2de205b0f63d") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:01.664655 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.664636 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv"] Oct 30 18:45:01.665037 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:01.665015 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d5532c_91de_471d_bdd7_e9ad1b0ce399.slice/crio-6c476791f158f01c19e69f56b050718c10d2a9a1341926d26d71accb735a8420 WatchSource:0}: Error finding container 6c476791f158f01c19e69f56b050718c10d2a9a1341926d26d71accb735a8420: Status 404 returned error can't find the container with id 6c476791f158f01c19e69f56b050718c10d2a9a1341926d26d71accb735a8420 Oct 30 18:45:01.667771 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.667748 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk"] Oct 30 18:45:01.668211 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:01.668192 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf488c3_95ac_4149_9b8b_ebb4522cc909.slice/crio-a5cc852637e6081b4a7537edc3422dce921e900c8efef09be1697bc9cfa69136 WatchSource:0}: Error finding container a5cc852637e6081b4a7537edc3422dce921e900c8efef09be1697bc9cfa69136: Status 404 returned error can't find the container with id a5cc852637e6081b4a7537edc3422dce921e900c8efef09be1697bc9cfa69136 Oct 30 18:45:01.675014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.674459 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m"] Oct 30 18:45:01.680213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.680194 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2"] Oct 30 18:45:01.681223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.681202 4862 cpu_manager.go:414] "containerMap: removing container" podUID="288f283c-eff8-4ade-9b4b-6543280108a3" containerName="payload-processing" Oct 30 18:45:01.681223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.681224 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288f283c-eff8-4ade-9b4b-6543280108a3" containerName="payload-processing" Oct 30 18:45:01.681399 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.681384 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="288f283c-eff8-4ade-9b4b-6543280108a3" containerName="payload-processing" Oct 30 18:45:01.687126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.687108 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf"] Oct 30 18:45:01.687218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.687203 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.688923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.688902 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn"] Oct 30 18:45:01.701569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.701537 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv"] Oct 30 18:45:01.704395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.704370 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2"] Oct 30 18:45:01.735871 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:01.735835 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1bb6bd_ca03_45c2_b563_0f884658acd0.slice/crio-2e8d9182d7b1b39acac3279521cc253901eb9172c2e99dcfcc840c75a7256a24 WatchSource:0}: Error finding container 2e8d9182d7b1b39acac3279521cc253901eb9172c2e99dcfcc840c75a7256a24: Status 404 returned error can't find the container with id 2e8d9182d7b1b39acac3279521cc253901eb9172c2e99dcfcc840c75a7256a24 Oct 30 18:45:01.836212 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:01.836183 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice/crio-f7e5d04da28a047904a7d95f1b4847a4f46dcbb5d0c76287ae0dedb583d85feb WatchSource:0}: Error finding container f7e5d04da28a047904a7d95f1b4847a4f46dcbb5d0c76287ae0dedb583d85feb: Status 404 returned error can't find the container with id f7e5d04da28a047904a7d95f1b4847a4f46dcbb5d0c76287ae0dedb583d85feb Oct 30 18:45:01.837480 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:01.837455 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c951a41_65ad_48dd_bb7e_1cfde1ef853f.slice/crio-39da8ea41ac306c48dd1c06c4d5eaf75aecc7cf58c76340cb7ac0c4eb2c37980 WatchSource:0}: Error finding container 39da8ea41ac306c48dd1c06c4d5eaf75aecc7cf58c76340cb7ac0c4eb2c37980: Status 404 returned error can't find the container with id 39da8ea41ac306c48dd1c06c4d5eaf75aecc7cf58c76340cb7ac0c4eb2c37980 Oct 30 18:45:01.872492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.872454 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5vwq\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-kube-api-access-p5vwq\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.872671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.872516 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.872796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.872765 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.873097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.873069 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.874108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.874082 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" event={"ID":"dc189cb4-8886-479c-903a-752c5ad19901","Type":"ContainerStarted","Data":"d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef"} Oct 30 18:45:01.875503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.875480 4862 generic.go:356] "Generic (PLEG): container finished" podID="288f283c-eff8-4ade-9b4b-6543280108a3" containerID="f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8" exitCode=0 Oct 30 18:45:01.875616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.875546 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" event={"ID":"288f283c-eff8-4ade-9b4b-6543280108a3","Type":"ContainerDied","Data":"f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8"} Oct 30 18:45:01.875616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.875556 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" Oct 30 18:45:01.875616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.875589 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2" event={"ID":"288f283c-eff8-4ade-9b4b-6543280108a3","Type":"ContainerDied","Data":"c99ad38f9a2a800f43d0e6c0836d0bf3b609d5226202d19c06d070daf2c5c984"} Oct 30 18:45:01.875778 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.875620 4862 scope.go:122] "RemoveContainer" containerID="f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8" Oct 30 18:45:01.877422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.877399 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" event={"ID":"e4d5532c-91de-471d-bdd7-e9ad1b0ce399","Type":"ContainerStarted","Data":"6c476791f158f01c19e69f56b050718c10d2a9a1341926d26d71accb735a8420"} Oct 30 18:45:01.878890 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.878867 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" event={"ID":"a97dc6f5-10b4-4d60-90bf-423a8da7ab74","Type":"ContainerStarted","Data":"15c286bfd3d4ef561ac5e1188805d15fa28e5f844beb702f44fa8d05f5748ad9"} Oct 30 18:45:01.880218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.880194 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" event={"ID":"dbf488c3-95ac-4149-9b8b-ebb4522cc909","Type":"ContainerStarted","Data":"a5cc852637e6081b4a7537edc3422dce921e900c8efef09be1697bc9cfa69136"} Oct 30 18:45:01.881548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.881524 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" event={"ID":"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca","Type":"ContainerStarted","Data":"ac05f999df790f60d350f6323755984962300a54efde3376a14ac73dd5377022"} Oct 30 18:45:01.882738 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.882716 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" event={"ID":"1ac031fb-b03c-4a94-9623-e19a1f4296e5","Type":"ContainerStarted","Data":"5f19654f06a658e6e9c50925eeba07fa8d2c33465f1ae03265a9c5bb474eacaa"} Oct 30 18:45:01.884146 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.884099 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" event={"ID":"edb49158-4950-4fcb-9ddc-aaa5005a019f","Type":"ContainerStarted","Data":"695c1689ba9e25855b4e2eade12650042396b2ca311f1a3b656108d37f22f574"} Oct 30 18:45:01.885706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.885685 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" event={"ID":"bb2168db-5894-4b73-ba78-754f1d8863bc","Type":"ContainerStarted","Data":"f7e5d04da28a047904a7d95f1b4847a4f46dcbb5d0c76287ae0dedb583d85feb"} Oct 30 18:45:01.887035 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.887016 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" event={"ID":"8c1bb6bd-ca03-45c2-b563-0f884658acd0","Type":"ContainerStarted","Data":"2e8d9182d7b1b39acac3279521cc253901eb9172c2e99dcfcc840c75a7256a24"} Oct 30 18:45:01.888314 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.888249 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" event={"ID":"0c951a41-65ad-48dd-bb7e-1cfde1ef853f","Type":"ContainerStarted","Data":"39da8ea41ac306c48dd1c06c4d5eaf75aecc7cf58c76340cb7ac0c4eb2c37980"} Oct 30 18:45:01.889601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.889580 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" event={"ID":"06cf935f-86dc-4790-93b2-c7baea9b6a42","Type":"ContainerStarted","Data":"d86b2a47ad26d6522adaaf0033c5ad076fa965e50695d07941117542e75c7bcc"} Oct 30 18:45:01.891143 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.891122 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" event={"ID":"cfffb210-225d-4707-9bdd-4608be2dc749","Type":"ContainerStarted","Data":"526ee3ef8ec437aef31a0423005f6e715f1b230a7d049f5a19c7592493e10914"} Oct 30 18:45:01.891228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.891158 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" event={"ID":"cfffb210-225d-4707-9bdd-4608be2dc749","Type":"ContainerStarted","Data":"1e724f0a43da2cea67e5b95df1f5debaf643fd29a1ce1ec71ce5cdf51b5eb645"} Oct 30 18:45:01.927002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.926968 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2"] Oct 30 18:45:01.932689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.932665 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-c85f2tqg2"] Oct 30 18:45:01.944347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.944321 4862 scope.go:122] "RemoveContainer" containerID="f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8" Oct 30 18:45:01.944654 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:01.944626 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8\": container with ID starting with f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8 not found: ID does not exist" containerID="f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8" Oct 30 18:45:01.944732 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.944671 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8"} err="failed to get container status \"f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8\": rpc error: code = NotFound desc = could not find container \"f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8\": container with ID starting with f331aaff1c25946f2fd213bd89d4e0e28d2e3b7db9b442871314581b953481e8 not found: ID does not exist" Oct 30 18:45:01.971578 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.971520 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" podStartSLOduration=1.9715059799999999 podStartE2EDuration="1.97150598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:01.968346842 +0000 UTC m=+2801.017391755" watchObservedRunningTime="2026-10-30 18:45:01.97150598 +0000 UTC m=+2801.020550890" Oct 30 18:45:01.974541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.974517 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.974672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.974659 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.974733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.974716 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5vwq\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-kube-api-access-p5vwq\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.974788 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.974771 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.975546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.975528 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:01.976257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:01.976232 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:02.029757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.029685 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:02.029923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.029899 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5vwq\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-kube-api-access-p5vwq\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:02.302182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.302155 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:02.583643 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:02.583619 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:02.583643 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:02.583642 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:02.583790 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:02.583689 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials podName:d04094c2-6fe0-4ae1-a731-b8679713e79f nodeName:}" failed. No retries permitted until 2026-10-30 18:45:04.583676538 +0000 UTC m=+2803.632721428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" (UID: "d04094c2-6fe0-4ae1-a731-b8679713e79f") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:02.650787 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.650761 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2"] Oct 30 18:45:02.684305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.684256 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:02.684421 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:02.684354 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:02.684421 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:02.684374 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:02.684421 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:02.684422 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials podName:f5920a67-84ce-4ede-896e-2de205b0f63d nodeName:}" failed. No retries permitted until 2026-10-30 18:45:04.684407395 +0000 UTC m=+2803.733452286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" (UID: "f5920a67-84ce-4ede-896e-2de205b0f63d") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:02.900157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.900125 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" event={"ID":"a97dc6f5-10b4-4d60-90bf-423a8da7ab74","Type":"ContainerStarted","Data":"01c658909bce5219891564d965d0ad4ae3614bc0ee7a84e38f7ab0bede954798"} Oct 30 18:45:02.901892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.901867 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" event={"ID":"dbf488c3-95ac-4149-9b8b-ebb4522cc909","Type":"ContainerStarted","Data":"188ae6f628bae144bacc0480b5b2bcb7162e3dae42daced561db481c3eec874c"} Oct 30 18:45:02.903504 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.903478 4862 generic.go:356] "Generic (PLEG): container finished" podID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerID="1829529423b15d019e052d8f7c7032e6c0cf71eef04c64c1ba45bd83649b7db3" exitCode=60 Oct 30 18:45:02.903604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.903546 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" event={"ID":"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca","Type":"ContainerDied","Data":"1829529423b15d019e052d8f7c7032e6c0cf71eef04c64c1ba45bd83649b7db3"} Oct 30 18:45:02.903792 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.903771 4862 scope.go:122] "RemoveContainer" containerID="1829529423b15d019e052d8f7c7032e6c0cf71eef04c64c1ba45bd83649b7db3" Oct 30 18:45:02.905394 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.905256 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" event={"ID":"bb2fed97-f53d-4f11-a967-8b975837a13e","Type":"ContainerStarted","Data":"45d653c7c8aa441158641e2dac4a9d0469d16e35c3b1adb47e1720dcff409c11"} Oct 30 18:45:02.905394 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.905315 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" event={"ID":"bb2fed97-f53d-4f11-a967-8b975837a13e","Type":"ContainerStarted","Data":"c6da7f1ead1d8103617c0e12794a26f27a46a094e4ce80d52c246433ee8b133a"} Oct 30 18:45:02.906929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.906903 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" event={"ID":"1ac031fb-b03c-4a94-9623-e19a1f4296e5","Type":"ContainerStarted","Data":"11d7cf515b8506db7829b4aa6960563a7467b6c15a0d2355298f8d56ee28e3e8"} Oct 30 18:45:02.908679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.908649 4862 generic.go:356] "Generic (PLEG): container finished" podID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerID="6c9f1aed1073838e5ff5c05e331d12a3ba1f76db6829d3c0aa0f738663c20ad6" exitCode=60 Oct 30 18:45:02.908780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.908692 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" event={"ID":"edb49158-4950-4fcb-9ddc-aaa5005a019f","Type":"ContainerDied","Data":"6c9f1aed1073838e5ff5c05e331d12a3ba1f76db6829d3c0aa0f738663c20ad6"} Oct 30 18:45:02.908931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.908915 4862 scope.go:122] "RemoveContainer" containerID="6c9f1aed1073838e5ff5c05e331d12a3ba1f76db6829d3c0aa0f738663c20ad6" Oct 30 18:45:02.910620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.910597 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" event={"ID":"bb2168db-5894-4b73-ba78-754f1d8863bc","Type":"ContainerStarted","Data":"1d2fbc53a936c489bdf4f09af7522f212db8df949ce9f63f233eb15b1aed33b0"} Oct 30 18:45:02.912214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.912189 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" event={"ID":"8c1bb6bd-ca03-45c2-b563-0f884658acd0","Type":"ContainerStarted","Data":"5877b2c5ea0c80ca3665a91685c61d2510bf11842082ac28ab2de7ac92556db4"} Oct 30 18:45:02.914155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.914118 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" event={"ID":"0c951a41-65ad-48dd-bb7e-1cfde1ef853f","Type":"ContainerStarted","Data":"802252905c2f5e727110863e97213f474e307d30d69f1f07eb204cfda48bb708"} Oct 30 18:45:02.915980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.915959 4862 generic.go:356] "Generic (PLEG): container finished" podID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerID="d3d91e4ee1b5b901cf24d3b790d8a0d99c6eb520513d3d3873cd1da933e39dd9" exitCode=60 Oct 30 18:45:02.916069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.916031 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" event={"ID":"06cf935f-86dc-4790-93b2-c7baea9b6a42","Type":"ContainerDied","Data":"d3d91e4ee1b5b901cf24d3b790d8a0d99c6eb520513d3d3873cd1da933e39dd9"} Oct 30 18:45:02.916231 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.916214 4862 scope.go:122] "RemoveContainer" containerID="d3d91e4ee1b5b901cf24d3b790d8a0d99c6eb520513d3d3873cd1da933e39dd9" Oct 30 18:45:02.917785 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.917692 4862 generic.go:356] "Generic (PLEG): container finished" podID="cfffb210-225d-4707-9bdd-4608be2dc749" containerID="526ee3ef8ec437aef31a0423005f6e715f1b230a7d049f5a19c7592493e10914" exitCode=60 Oct 30 18:45:02.917785 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.917767 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" event={"ID":"cfffb210-225d-4707-9bdd-4608be2dc749","Type":"ContainerDied","Data":"526ee3ef8ec437aef31a0423005f6e715f1b230a7d049f5a19c7592493e10914"} Oct 30 18:45:02.917983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.917933 4862 scope.go:122] "RemoveContainer" containerID="526ee3ef8ec437aef31a0423005f6e715f1b230a7d049f5a19c7592493e10914" Oct 30 18:45:02.919418 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.919394 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" event={"ID":"7bfe54c8-1ba5-422b-96b0-2f4a846e083e","Type":"ContainerStarted","Data":"537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120"} Oct 30 18:45:02.919518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.919438 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" event={"ID":"7bfe54c8-1ba5-422b-96b0-2f4a846e083e","Type":"ContainerStarted","Data":"eaabd594e44d7c4d161ed1c44e720f5a1a33088226b66bcfa439361126a4bd18"} Oct 30 18:45:02.919518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.919484 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:02.920995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.920974 4862 generic.go:356] "Generic (PLEG): container finished" podID="dc189cb4-8886-479c-903a-752c5ad19901" containerID="f5a96234a1b2d0277ae07ca086130dc4e2df1fa988aa0cee6ed9c2f3179ea004" exitCode=60 Oct 30 18:45:02.921097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.921054 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" event={"ID":"dc189cb4-8886-479c-903a-752c5ad19901","Type":"ContainerDied","Data":"f5a96234a1b2d0277ae07ca086130dc4e2df1fa988aa0cee6ed9c2f3179ea004"} Oct 30 18:45:02.921232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.921217 4862 scope.go:122] "RemoveContainer" containerID="f5a96234a1b2d0277ae07ca086130dc4e2df1fa988aa0cee6ed9c2f3179ea004" Oct 30 18:45:02.923491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.923470 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" event={"ID":"e4d5532c-91de-471d-bdd7-e9ad1b0ce399","Type":"ContainerStarted","Data":"a7c5e517313f33357d6fd423a339429a19d3583193bd802f2d1e81e8277e3140"} Oct 30 18:45:02.939942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.939876 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" podStartSLOduration=2.939859712 podStartE2EDuration="2.939859712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:02.938625858 +0000 UTC m=+2801.987670772" watchObservedRunningTime="2026-10-30 18:45:02.939859712 +0000 UTC m=+2801.988904626" Oct 30 18:45:02.977929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:02.977881 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" podStartSLOduration=2.977866521 podStartE2EDuration="2.977866521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:02.976577848 +0000 UTC m=+2802.025622766" watchObservedRunningTime="2026-10-30 18:45:02.977866521 +0000 UTC m=+2802.026911430" Oct 30 18:45:03.018121 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.018075 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" podStartSLOduration=3.018061078 podStartE2EDuration="3.018061078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.015150429 +0000 UTC m=+2802.064195343" watchObservedRunningTime="2026-10-30 18:45:03.018061078 +0000 UTC m=+2802.067106011" Oct 30 18:45:03.055150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.055108 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" podStartSLOduration=2.055093736 podStartE2EDuration="2.055093736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.053990656 +0000 UTC m=+2802.103035588" watchObservedRunningTime="2026-10-30 18:45:03.055093736 +0000 UTC m=+2802.104138642" Oct 30 18:45:03.064132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.064108 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288f283c-eff8-4ade-9b4b-6543280108a3" path="/var/lib/kubelet/pods/288f283c-eff8-4ade-9b4b-6543280108a3/volumes" Oct 30 18:45:03.150482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.150442 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" podStartSLOduration=3.150428681 podStartE2EDuration="3.150428681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.148012011 +0000 UTC m=+2802.197056934" watchObservedRunningTime="2026-10-30 18:45:03.150428681 +0000 UTC m=+2802.199473592" Oct 30 18:45:03.194973 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.194929 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" podStartSLOduration=3.194916231 podStartE2EDuration="3.194916231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.192463324 +0000 UTC m=+2802.241508240" watchObservedRunningTime="2026-10-30 18:45:03.194916231 +0000 UTC m=+2802.243961144" Oct 30 18:45:03.238591 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.238534 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" podStartSLOduration=3.238516255 podStartE2EDuration="3.238516255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.236245032 +0000 UTC m=+2802.285289946" watchObservedRunningTime="2026-10-30 18:45:03.238516255 +0000 UTC m=+2802.287561172" Oct 30 18:45:03.273730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.273680 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" podStartSLOduration=3.273666983 podStartE2EDuration="3.273666983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.27266223 +0000 UTC m=+2802.321707142" watchObservedRunningTime="2026-10-30 18:45:03.273666983 +0000 UTC m=+2802.322711895" Oct 30 18:45:03.464038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.463982 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" podStartSLOduration=3.463965481 podStartE2EDuration="3.463965481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.462457729 +0000 UTC m=+2802.511502642" watchObservedRunningTime="2026-10-30 18:45:03.463965481 +0000 UTC m=+2802.513010395" Oct 30 18:45:03.933884 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.933795 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" event={"ID":"dc189cb4-8886-479c-903a-752c5ad19901","Type":"ContainerStarted","Data":"18cedf1c89f5dba79dae9e100936e3587a52fe6481ce3d112e2f1605c490cb7e"} Oct 30 18:45:03.935392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.935362 4862 generic.go:356] "Generic (PLEG): container finished" podID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerID="a7c5e517313f33357d6fd423a339429a19d3583193bd802f2d1e81e8277e3140" exitCode=60 Oct 30 18:45:03.935523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.935439 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" event={"ID":"e4d5532c-91de-471d-bdd7-e9ad1b0ce399","Type":"ContainerDied","Data":"a7c5e517313f33357d6fd423a339429a19d3583193bd802f2d1e81e8277e3140"} Oct 30 18:45:03.935645 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.935629 4862 scope.go:122] "RemoveContainer" containerID="a7c5e517313f33357d6fd423a339429a19d3583193bd802f2d1e81e8277e3140" Oct 30 18:45:03.937122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.937095 4862 generic.go:356] "Generic (PLEG): container finished" podID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerID="01c658909bce5219891564d965d0ad4ae3614bc0ee7a84e38f7ab0bede954798" exitCode=60 Oct 30 18:45:03.937251 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.937173 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" event={"ID":"a97dc6f5-10b4-4d60-90bf-423a8da7ab74","Type":"ContainerDied","Data":"01c658909bce5219891564d965d0ad4ae3614bc0ee7a84e38f7ab0bede954798"} Oct 30 18:45:03.937421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.937387 4862 scope.go:122] "RemoveContainer" containerID="01c658909bce5219891564d965d0ad4ae3614bc0ee7a84e38f7ab0bede954798" Oct 30 18:45:03.939083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.939065 4862 generic.go:356] "Generic (PLEG): container finished" podID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerID="188ae6f628bae144bacc0480b5b2bcb7162e3dae42daced561db481c3eec874c" exitCode=60 Oct 30 18:45:03.939172 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.939126 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" event={"ID":"dbf488c3-95ac-4149-9b8b-ebb4522cc909","Type":"ContainerDied","Data":"188ae6f628bae144bacc0480b5b2bcb7162e3dae42daced561db481c3eec874c"} Oct 30 18:45:03.939477 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.939424 4862 scope.go:122] "RemoveContainer" containerID="188ae6f628bae144bacc0480b5b2bcb7162e3dae42daced561db481c3eec874c" Oct 30 18:45:03.941059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.941040 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" event={"ID":"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca","Type":"ContainerStarted","Data":"814af0f682351c6f0624c61703cea6234d7eb9da1f8396f677339d2cc1a7a5f2"} Oct 30 18:45:03.942665 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.942645 4862 generic.go:356] "Generic (PLEG): container finished" podID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerID="45d653c7c8aa441158641e2dac4a9d0469d16e35c3b1adb47e1720dcff409c11" exitCode=60 Oct 30 18:45:03.942766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.942718 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" event={"ID":"bb2fed97-f53d-4f11-a967-8b975837a13e","Type":"ContainerDied","Data":"45d653c7c8aa441158641e2dac4a9d0469d16e35c3b1adb47e1720dcff409c11"} Oct 30 18:45:03.942890 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.942878 4862 scope.go:122] "RemoveContainer" containerID="45d653c7c8aa441158641e2dac4a9d0469d16e35c3b1adb47e1720dcff409c11" Oct 30 18:45:03.944230 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.944211 4862 generic.go:356] "Generic (PLEG): container finished" podID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerID="11d7cf515b8506db7829b4aa6960563a7467b6c15a0d2355298f8d56ee28e3e8" exitCode=60 Oct 30 18:45:03.944365 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.944292 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" event={"ID":"1ac031fb-b03c-4a94-9623-e19a1f4296e5","Type":"ContainerDied","Data":"11d7cf515b8506db7829b4aa6960563a7467b6c15a0d2355298f8d56ee28e3e8"} Oct 30 18:45:03.944479 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.944463 4862 scope.go:122] "RemoveContainer" containerID="11d7cf515b8506db7829b4aa6960563a7467b6c15a0d2355298f8d56ee28e3e8" Oct 30 18:45:03.946257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.946232 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" event={"ID":"edb49158-4950-4fcb-9ddc-aaa5005a019f","Type":"ContainerStarted","Data":"74210ddc56ddad30bcb6f6622a43233c58726a94f1fdbbfeb0b9db435689ccc7"} Oct 30 18:45:03.948202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.948066 4862 generic.go:356] "Generic (PLEG): container finished" podID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerID="1d2fbc53a936c489bdf4f09af7522f212db8df949ce9f63f233eb15b1aed33b0" exitCode=60 Oct 30 18:45:03.948352 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.948332 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" event={"ID":"bb2168db-5894-4b73-ba78-754f1d8863bc","Type":"ContainerDied","Data":"1d2fbc53a936c489bdf4f09af7522f212db8df949ce9f63f233eb15b1aed33b0"} Oct 30 18:45:03.948466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.948415 4862 scope.go:122] "RemoveContainer" containerID="1d2fbc53a936c489bdf4f09af7522f212db8df949ce9f63f233eb15b1aed33b0" Oct 30 18:45:03.949857 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.949838 4862 generic.go:356] "Generic (PLEG): container finished" podID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerID="5877b2c5ea0c80ca3665a91685c61d2510bf11842082ac28ab2de7ac92556db4" exitCode=60 Oct 30 18:45:03.949964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.949916 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" event={"ID":"8c1bb6bd-ca03-45c2-b563-0f884658acd0","Type":"ContainerDied","Data":"5877b2c5ea0c80ca3665a91685c61d2510bf11842082ac28ab2de7ac92556db4"} Oct 30 18:45:03.950076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.950062 4862 scope.go:122] "RemoveContainer" containerID="5877b2c5ea0c80ca3665a91685c61d2510bf11842082ac28ab2de7ac92556db4" Oct 30 18:45:03.951613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.951592 4862 generic.go:356] "Generic (PLEG): container finished" podID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerID="802252905c2f5e727110863e97213f474e307d30d69f1f07eb204cfda48bb708" exitCode=60 Oct 30 18:45:03.951699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.951657 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" event={"ID":"0c951a41-65ad-48dd-bb7e-1cfde1ef853f","Type":"ContainerDied","Data":"802252905c2f5e727110863e97213f474e307d30d69f1f07eb204cfda48bb708"} Oct 30 18:45:03.951878 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.951859 4862 scope.go:122] "RemoveContainer" containerID="802252905c2f5e727110863e97213f474e307d30d69f1f07eb204cfda48bb708" Oct 30 18:45:03.953796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.953772 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" event={"ID":"06cf935f-86dc-4790-93b2-c7baea9b6a42","Type":"ContainerStarted","Data":"32ef75be542dc2c2e329262fc6d86c7c5225cf892883120dad9cb1af98217abd"} Oct 30 18:45:03.956854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.956593 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" event={"ID":"cfffb210-225d-4707-9bdd-4608be2dc749","Type":"ContainerStarted","Data":"b7239eb43ffd66ddadd16da758b244cb780a9a25b3eaf76decd88659a175c4ca"} Oct 30 18:45:03.973444 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:03.973400 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" podStartSLOduration=3.973385186 podStartE2EDuration="3.973385186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.970792889 +0000 UTC m=+2803.019837985" watchObservedRunningTime="2026-10-30 18:45:03.973385186 +0000 UTC m=+2803.022430099" Oct 30 18:45:04.125202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.125150 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" podStartSLOduration=4.125135927 podStartE2EDuration="4.125135927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:04.122456381 +0000 UTC m=+2803.171501294" watchObservedRunningTime="2026-10-30 18:45:04.125135927 +0000 UTC m=+2803.174180840" Oct 30 18:45:04.366187 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.366140 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" podStartSLOduration=4.366125495 podStartE2EDuration="4.366125495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:04.364859559 +0000 UTC m=+2803.413904472" watchObservedRunningTime="2026-10-30 18:45:04.366125495 +0000 UTC m=+2803.415170408" Oct 30 18:45:04.443415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.443360 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" podStartSLOduration=4.443344718 podStartE2EDuration="4.443344718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:04.439348898 +0000 UTC m=+2803.488393816" watchObservedRunningTime="2026-10-30 18:45:04.443344718 +0000 UTC m=+2803.492389630" Oct 30 18:45:04.611214 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.611189 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:04.611214 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.611211 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:04.611559 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.611256 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials podName:d04094c2-6fe0-4ae1-a731-b8679713e79f nodeName:}" failed. No retries permitted until 2026-10-30 18:45:08.611242669 +0000 UTC m=+2807.660287559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" (UID: "d04094c2-6fe0-4ae1-a731-b8679713e79f") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:04.712347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.712323 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:04.712477 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.712459 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:04.712530 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.712485 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:04.712530 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.712526 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials podName:f5920a67-84ce-4ede-896e-2de205b0f63d nodeName:}" failed. No retries permitted until 2026-10-30 18:45:08.712513628 +0000 UTC m=+2807.761558538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" (UID: "f5920a67-84ce-4ede-896e-2de205b0f63d") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:04.965426 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.965347 4862 generic.go:356] "Generic (PLEG): container finished" podID="dc189cb4-8886-479c-903a-752c5ad19901" containerID="18cedf1c89f5dba79dae9e100936e3587a52fe6481ce3d112e2f1605c490cb7e" exitCode=60 Oct 30 18:45:04.965569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.965435 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" event={"ID":"dc189cb4-8886-479c-903a-752c5ad19901","Type":"ContainerDied","Data":"18cedf1c89f5dba79dae9e100936e3587a52fe6481ce3d112e2f1605c490cb7e"} Oct 30 18:45:04.965569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.965509 4862 scope.go:122] "RemoveContainer" containerID="f5a96234a1b2d0277ae07ca086130dc4e2df1fa988aa0cee6ed9c2f3179ea004" Oct 30 18:45:04.965748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.965735 4862 scope.go:122] "RemoveContainer" containerID="18cedf1c89f5dba79dae9e100936e3587a52fe6481ce3d112e2f1605c490cb7e" Oct 30 18:45:04.966022 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.966005 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg_odh-ai-gateway-infra(dc189cb4-8886-479c-903a-752c5ad19901)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" podUID="dc189cb4-8886-479c-903a-752c5ad19901" Oct 30 18:45:04.968109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.968076 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" event={"ID":"e4d5532c-91de-471d-bdd7-e9ad1b0ce399","Type":"ContainerStarted","Data":"520f430d301a9b4a4793db8f6234b7a25f2acca92a2819fd6db4a8b94f7272c8"} Oct 30 18:45:04.970064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.970044 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" event={"ID":"a97dc6f5-10b4-4d60-90bf-423a8da7ab74","Type":"ContainerStarted","Data":"182d755e882bc359c49288be1a6fde2a835d0027a2f3e9742b33d882879a6c3b"} Oct 30 18:45:04.972244 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.972220 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" event={"ID":"dbf488c3-95ac-4149-9b8b-ebb4522cc909","Type":"ContainerStarted","Data":"1dc1cb2f01a15be7d2475051a0722bef96900d7d3c42884c95e22ce5bf21639e"} Oct 30 18:45:04.974191 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.974166 4862 generic.go:356] "Generic (PLEG): container finished" podID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerID="814af0f682351c6f0624c61703cea6234d7eb9da1f8396f677339d2cc1a7a5f2" exitCode=60 Oct 30 18:45:04.974291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.974214 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" event={"ID":"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca","Type":"ContainerDied","Data":"814af0f682351c6f0624c61703cea6234d7eb9da1f8396f677339d2cc1a7a5f2"} Oct 30 18:45:04.974459 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.974439 4862 scope.go:122] "RemoveContainer" containerID="814af0f682351c6f0624c61703cea6234d7eb9da1f8396f677339d2cc1a7a5f2" Oct 30 18:45:04.974680 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.974656 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc_odh-ai-gateway-infra(b50bc46b-93f1-40f9-a6e3-78bf998aa2ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" Oct 30 18:45:04.976197 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.976178 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" event={"ID":"bb2fed97-f53d-4f11-a967-8b975837a13e","Type":"ContainerStarted","Data":"0113968fcbe0fdbe9bd293bb7bef053fd6195d086f24f8ba05a1d3a7d9c77c17"} Oct 30 18:45:04.983032 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.983012 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" event={"ID":"1ac031fb-b03c-4a94-9623-e19a1f4296e5","Type":"ContainerStarted","Data":"9dd2808e445e7ec566a48a20fce7d1a5137034aa6efa14748e8c9e7659628835"} Oct 30 18:45:04.984357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.984342 4862 scope.go:122] "RemoveContainer" containerID="1829529423b15d019e052d8f7c7032e6c0cf71eef04c64c1ba45bd83649b7db3" Oct 30 18:45:04.985401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.985382 4862 generic.go:356] "Generic (PLEG): container finished" podID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerID="74210ddc56ddad30bcb6f6622a43233c58726a94f1fdbbfeb0b9db435689ccc7" exitCode=60 Oct 30 18:45:04.985481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.985463 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" event={"ID":"edb49158-4950-4fcb-9ddc-aaa5005a019f","Type":"ContainerDied","Data":"74210ddc56ddad30bcb6f6622a43233c58726a94f1fdbbfeb0b9db435689ccc7"} Oct 30 18:45:04.985694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.985671 4862 scope.go:122] "RemoveContainer" containerID="74210ddc56ddad30bcb6f6622a43233c58726a94f1fdbbfeb0b9db435689ccc7" Oct 30 18:45:04.985966 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.985940 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z_odh-ai-gateway-infra(edb49158-4950-4fcb-9ddc-aaa5005a019f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" Oct 30 18:45:04.987301 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.987280 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" event={"ID":"bb2168db-5894-4b73-ba78-754f1d8863bc","Type":"ContainerStarted","Data":"06beb9bb9d0d2d0f6b229e4994f2f0f0a5cc89fa60f0b03fd7b1ab287f2828c7"} Oct 30 18:45:04.988900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.988882 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" event={"ID":"8c1bb6bd-ca03-45c2-b563-0f884658acd0","Type":"ContainerStarted","Data":"77d195cfaffc0e3faa922a4c7d70ad05110ef61681302b62dbf76070a127b6f0"} Oct 30 18:45:04.990776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.990757 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" event={"ID":"0c951a41-65ad-48dd-bb7e-1cfde1ef853f","Type":"ContainerStarted","Data":"3c92dfe2bd5e206e39170470662e77300916138d1c95ad20d0804f6d47414ce3"} Oct 30 18:45:04.992607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.992584 4862 generic.go:356] "Generic (PLEG): container finished" podID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerID="32ef75be542dc2c2e329262fc6d86c7c5225cf892883120dad9cb1af98217abd" exitCode=60 Oct 30 18:45:04.992706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.992655 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" event={"ID":"06cf935f-86dc-4790-93b2-c7baea9b6a42","Type":"ContainerDied","Data":"32ef75be542dc2c2e329262fc6d86c7c5225cf892883120dad9cb1af98217abd"} Oct 30 18:45:04.992837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.992820 4862 scope.go:122] "RemoveContainer" containerID="32ef75be542dc2c2e329262fc6d86c7c5225cf892883120dad9cb1af98217abd" Oct 30 18:45:04.993071 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.993046 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww_odh-ai-gateway-infra(06cf935f-86dc-4790-93b2-c7baea9b6a42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" Oct 30 18:45:04.994556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.994536 4862 generic.go:356] "Generic (PLEG): container finished" podID="cfffb210-225d-4707-9bdd-4608be2dc749" containerID="b7239eb43ffd66ddadd16da758b244cb780a9a25b3eaf76decd88659a175c4ca" exitCode=60 Oct 30 18:45:04.994640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.994559 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" event={"ID":"cfffb210-225d-4707-9bdd-4608be2dc749","Type":"ContainerDied","Data":"b7239eb43ffd66ddadd16da758b244cb780a9a25b3eaf76decd88659a175c4ca"} Oct 30 18:45:04.994788 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.994751 4862 scope.go:122] "RemoveContainer" containerID="b7239eb43ffd66ddadd16da758b244cb780a9a25b3eaf76decd88659a175c4ca" Oct 30 18:45:04.994942 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:04.994924 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm_odh-ai-gateway-infra(cfffb210-225d-4707-9bdd-4608be2dc749)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" Oct 30 18:45:04.998734 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:04.998715 4862 scope.go:122] "RemoveContainer" containerID="6c9f1aed1073838e5ff5c05e331d12a3ba1f76db6829d3c0aa0f738663c20ad6" Oct 30 18:45:05.012742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:05.012724 4862 scope.go:122] "RemoveContainer" containerID="d3d91e4ee1b5b901cf24d3b790d8a0d99c6eb520513d3d3873cd1da933e39dd9" Oct 30 18:45:05.025921 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:05.025904 4862 scope.go:122] "RemoveContainer" containerID="526ee3ef8ec437aef31a0423005f6e715f1b230a7d049f5a19c7592493e10914" Oct 30 18:45:06.004077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.004043 4862 scope.go:122] "RemoveContainer" containerID="18cedf1c89f5dba79dae9e100936e3587a52fe6481ce3d112e2f1605c490cb7e" Oct 30 18:45:06.004594 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.004439 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg_odh-ai-gateway-infra(dc189cb4-8886-479c-903a-752c5ad19901)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" podUID="dc189cb4-8886-479c-903a-752c5ad19901" Oct 30 18:45:06.013539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.006127 4862 generic.go:356] "Generic (PLEG): container finished" podID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerID="520f430d301a9b4a4793db8f6234b7a25f2acca92a2819fd6db4a8b94f7272c8" exitCode=60 Oct 30 18:45:06.013539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.006248 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" event={"ID":"e4d5532c-91de-471d-bdd7-e9ad1b0ce399","Type":"ContainerDied","Data":"520f430d301a9b4a4793db8f6234b7a25f2acca92a2819fd6db4a8b94f7272c8"} Oct 30 18:45:06.013539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.006322 4862 scope.go:122] "RemoveContainer" containerID="a7c5e517313f33357d6fd423a339429a19d3583193bd802f2d1e81e8277e3140" Oct 30 18:45:06.013539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.006732 4862 scope.go:122] "RemoveContainer" containerID="520f430d301a9b4a4793db8f6234b7a25f2acca92a2819fd6db4a8b94f7272c8" Oct 30 18:45:06.013539 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.007062 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv_odh-ai-gateway-infra(e4d5532c-91de-471d-bdd7-e9ad1b0ce399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" Oct 30 18:45:06.031646 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.031613 4862 generic.go:356] "Generic (PLEG): container finished" podID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerID="182d755e882bc359c49288be1a6fde2a835d0027a2f3e9742b33d882879a6c3b" exitCode=60 Oct 30 18:45:06.031791 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.031722 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" event={"ID":"a97dc6f5-10b4-4d60-90bf-423a8da7ab74","Type":"ContainerDied","Data":"182d755e882bc359c49288be1a6fde2a835d0027a2f3e9742b33d882879a6c3b"} Oct 30 18:45:06.032551 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.032520 4862 scope.go:122] "RemoveContainer" containerID="182d755e882bc359c49288be1a6fde2a835d0027a2f3e9742b33d882879a6c3b" Oct 30 18:45:06.032917 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.032890 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24_odh-ai-gateway-infra(a97dc6f5-10b4-4d60-90bf-423a8da7ab74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" Oct 30 18:45:06.035020 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.034994 4862 generic.go:356] "Generic (PLEG): container finished" podID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerID="1dc1cb2f01a15be7d2475051a0722bef96900d7d3c42884c95e22ce5bf21639e" exitCode=60 Oct 30 18:45:06.035145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.035087 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" event={"ID":"dbf488c3-95ac-4149-9b8b-ebb4522cc909","Type":"ContainerDied","Data":"1dc1cb2f01a15be7d2475051a0722bef96900d7d3c42884c95e22ce5bf21639e"} Oct 30 18:45:06.035455 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.035436 4862 scope.go:122] "RemoveContainer" containerID="1dc1cb2f01a15be7d2475051a0722bef96900d7d3c42884c95e22ce5bf21639e" Oct 30 18:45:06.035723 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.035705 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk_odh-ai-gateway-infra(dbf488c3-95ac-4149-9b8b-ebb4522cc909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" Oct 30 18:45:06.037375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.037332 4862 scope.go:122] "RemoveContainer" containerID="814af0f682351c6f0624c61703cea6234d7eb9da1f8396f677339d2cc1a7a5f2" Oct 30 18:45:06.037708 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.037603 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc_odh-ai-gateway-infra(b50bc46b-93f1-40f9-a6e3-78bf998aa2ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" Oct 30 18:45:06.039868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.039836 4862 generic.go:356] "Generic (PLEG): container finished" podID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerID="0113968fcbe0fdbe9bd293bb7bef053fd6195d086f24f8ba05a1d3a7d9c77c17" exitCode=60 Oct 30 18:45:06.039960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.039919 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" event={"ID":"bb2fed97-f53d-4f11-a967-8b975837a13e","Type":"ContainerDied","Data":"0113968fcbe0fdbe9bd293bb7bef053fd6195d086f24f8ba05a1d3a7d9c77c17"} Oct 30 18:45:06.040136 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.040116 4862 scope.go:122] "RemoveContainer" containerID="0113968fcbe0fdbe9bd293bb7bef053fd6195d086f24f8ba05a1d3a7d9c77c17" Oct 30 18:45:06.040405 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.040383 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-dtlxv_odh-ai-gateway-infra(bb2fed97-f53d-4f11-a967-8b975837a13e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" Oct 30 18:45:06.041933 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.041913 4862 generic.go:356] "Generic (PLEG): container finished" podID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerID="9dd2808e445e7ec566a48a20fce7d1a5137034aa6efa14748e8c9e7659628835" exitCode=60 Oct 30 18:45:06.042005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.041976 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" event={"ID":"1ac031fb-b03c-4a94-9623-e19a1f4296e5","Type":"ContainerDied","Data":"9dd2808e445e7ec566a48a20fce7d1a5137034aa6efa14748e8c9e7659628835"} Oct 30 18:45:06.042188 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.042171 4862 scope.go:122] "RemoveContainer" containerID="9dd2808e445e7ec566a48a20fce7d1a5137034aa6efa14748e8c9e7659628835" Oct 30 18:45:06.042468 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.042444 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg_odh-ai-gateway-infra(1ac031fb-b03c-4a94-9623-e19a1f4296e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" Oct 30 18:45:06.044889 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.044870 4862 scope.go:122] "RemoveContainer" containerID="74210ddc56ddad30bcb6f6622a43233c58726a94f1fdbbfeb0b9db435689ccc7" Oct 30 18:45:06.047186 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.047156 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z_odh-ai-gateway-infra(edb49158-4950-4fcb-9ddc-aaa5005a019f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" Oct 30 18:45:06.048388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.048365 4862 generic.go:356] "Generic (PLEG): container finished" podID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerID="06beb9bb9d0d2d0f6b229e4994f2f0f0a5cc89fa60f0b03fd7b1ab287f2828c7" exitCode=60 Oct 30 18:45:06.048574 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.048548 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" event={"ID":"bb2168db-5894-4b73-ba78-754f1d8863bc","Type":"ContainerDied","Data":"06beb9bb9d0d2d0f6b229e4994f2f0f0a5cc89fa60f0b03fd7b1ab287f2828c7"} Oct 30 18:45:06.048838 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.048817 4862 scope.go:122] "RemoveContainer" containerID="06beb9bb9d0d2d0f6b229e4994f2f0f0a5cc89fa60f0b03fd7b1ab287f2828c7" Oct 30 18:45:06.049114 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.049054 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf_odh-ai-gateway-infra(bb2168db-5894-4b73-ba78-754f1d8863bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" Oct 30 18:45:06.049531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.049512 4862 scope.go:122] "RemoveContainer" containerID="01c658909bce5219891564d965d0ad4ae3614bc0ee7a84e38f7ab0bede954798" Oct 30 18:45:06.051603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.051576 4862 generic.go:356] "Generic (PLEG): container finished" podID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerID="77d195cfaffc0e3faa922a4c7d70ad05110ef61681302b62dbf76070a127b6f0" exitCode=60 Oct 30 18:45:06.051691 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.051614 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" event={"ID":"8c1bb6bd-ca03-45c2-b563-0f884658acd0","Type":"ContainerDied","Data":"77d195cfaffc0e3faa922a4c7d70ad05110ef61681302b62dbf76070a127b6f0"} Oct 30 18:45:06.051873 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.051857 4862 scope.go:122] "RemoveContainer" containerID="77d195cfaffc0e3faa922a4c7d70ad05110ef61681302b62dbf76070a127b6f0" Oct 30 18:45:06.052096 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.052076 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m_odh-ai-gateway-infra(8c1bb6bd-ca03-45c2-b563-0f884658acd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" Oct 30 18:45:06.053671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.053653 4862 generic.go:356] "Generic (PLEG): container finished" podID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerID="3c92dfe2bd5e206e39170470662e77300916138d1c95ad20d0804f6d47414ce3" exitCode=60 Oct 30 18:45:06.053778 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.053711 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" event={"ID":"0c951a41-65ad-48dd-bb7e-1cfde1ef853f","Type":"ContainerDied","Data":"3c92dfe2bd5e206e39170470662e77300916138d1c95ad20d0804f6d47414ce3"} Oct 30 18:45:06.053901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.053886 4862 scope.go:122] "RemoveContainer" containerID="3c92dfe2bd5e206e39170470662e77300916138d1c95ad20d0804f6d47414ce3" Oct 30 18:45:06.054110 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.054090 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn_odh-ai-gateway-infra(0c951a41-65ad-48dd-bb7e-1cfde1ef853f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" Oct 30 18:45:06.055605 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.055535 4862 scope.go:122] "RemoveContainer" containerID="32ef75be542dc2c2e329262fc6d86c7c5225cf892883120dad9cb1af98217abd" Oct 30 18:45:06.055768 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.055748 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww_odh-ai-gateway-infra(06cf935f-86dc-4790-93b2-c7baea9b6a42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" Oct 30 18:45:06.057613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.057597 4862 scope.go:122] "RemoveContainer" containerID="b7239eb43ffd66ddadd16da758b244cb780a9a25b3eaf76decd88659a175c4ca" Oct 30 18:45:06.057773 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:06.057757 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm_odh-ai-gateway-infra(cfffb210-225d-4707-9bdd-4608be2dc749)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" Oct 30 18:45:06.065001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.064987 4862 scope.go:122] "RemoveContainer" containerID="188ae6f628bae144bacc0480b5b2bcb7162e3dae42daced561db481c3eec874c" Oct 30 18:45:06.078190 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.078173 4862 scope.go:122] "RemoveContainer" containerID="45d653c7c8aa441158641e2dac4a9d0469d16e35c3b1adb47e1720dcff409c11" Oct 30 18:45:06.092050 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.092035 4862 scope.go:122] "RemoveContainer" containerID="11d7cf515b8506db7829b4aa6960563a7467b6c15a0d2355298f8d56ee28e3e8" Oct 30 18:45:06.105031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.105016 4862 scope.go:122] "RemoveContainer" containerID="1d2fbc53a936c489bdf4f09af7522f212db8df949ce9f63f233eb15b1aed33b0" Oct 30 18:45:06.120732 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.120711 4862 scope.go:122] "RemoveContainer" containerID="5877b2c5ea0c80ca3665a91685c61d2510bf11842082ac28ab2de7ac92556db4" Oct 30 18:45:06.134649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:06.134632 4862 scope.go:122] "RemoveContainer" containerID="802252905c2f5e727110863e97213f474e307d30d69f1f07eb204cfda48bb708" Oct 30 18:45:07.066523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:07.066491 4862 scope.go:122] "RemoveContainer" containerID="520f430d301a9b4a4793db8f6234b7a25f2acca92a2819fd6db4a8b94f7272c8" Oct 30 18:45:07.066882 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:07.066713 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv_odh-ai-gateway-infra(e4d5532c-91de-471d-bdd7-e9ad1b0ce399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" Oct 30 18:45:07.068362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:07.068341 4862 scope.go:122] "RemoveContainer" containerID="182d755e882bc359c49288be1a6fde2a835d0027a2f3e9742b33d882879a6c3b" Oct 30 18:45:07.068560 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:07.068540 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24_odh-ai-gateway-infra(a97dc6f5-10b4-4d60-90bf-423a8da7ab74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" Oct 30 18:45:07.069837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:07.069818 4862 scope.go:122] "RemoveContainer" containerID="1dc1cb2f01a15be7d2475051a0722bef96900d7d3c42884c95e22ce5bf21639e" Oct 30 18:45:07.070038 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:07.070015 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk_odh-ai-gateway-infra(dbf488c3-95ac-4149-9b8b-ebb4522cc909)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" Oct 30 18:45:07.071239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:07.071223 4862 scope.go:122] "RemoveContainer" containerID="0113968fcbe0fdbe9bd293bb7bef053fd6195d086f24f8ba05a1d3a7d9c77c17" Oct 30 18:45:07.071497 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:07.071472 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-dtlxv_odh-ai-gateway-infra(bb2fed97-f53d-4f11-a967-8b975837a13e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" Oct 30 18:45:07.072835 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:07.072819 4862 scope.go:122] "RemoveContainer" containerID="9dd2808e445e7ec566a48a20fce7d1a5137034aa6efa14748e8c9e7659628835" Oct 30 18:45:07.073021 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:07.073001 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg_odh-ai-gateway-infra(1ac031fb-b03c-4a94-9623-e19a1f4296e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" Oct 30 18:45:07.074202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:07.074187 4862 scope.go:122] "RemoveContainer" containerID="06beb9bb9d0d2d0f6b229e4994f2f0f0a5cc89fa60f0b03fd7b1ab287f2828c7" Oct 30 18:45:07.074424 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:07.074405 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf_odh-ai-gateway-infra(bb2168db-5894-4b73-ba78-754f1d8863bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" Oct 30 18:45:07.075578 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:07.075564 4862 scope.go:122] "RemoveContainer" containerID="77d195cfaffc0e3faa922a4c7d70ad05110ef61681302b62dbf76070a127b6f0" Oct 30 18:45:07.075777 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:07.075753 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m_odh-ai-gateway-infra(8c1bb6bd-ca03-45c2-b563-0f884658acd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" Oct 30 18:45:07.076948 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:07.076932 4862 scope.go:122] "RemoveContainer" containerID="3c92dfe2bd5e206e39170470662e77300916138d1c95ad20d0804f6d47414ce3" Oct 30 18:45:07.077119 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:07.077103 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn_odh-ai-gateway-infra(0c951a41-65ad-48dd-bb7e-1cfde1ef853f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" Oct 30 18:45:07.923930 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:07.923887 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" podUID="d04094c2-6fe0-4ae1-a731-b8679713e79f" containerName="payload-processing" containerID="cri-o://0dec8ee1a98d1008c3e60100da21df00224bcf304d3385f0099df24502954ab5" gracePeriod=25 Oct 30 18:45:08.085421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.085390 4862 generic.go:356] "Generic (PLEG): container finished" podID="d04094c2-6fe0-4ae1-a731-b8679713e79f" containerID="0dec8ee1a98d1008c3e60100da21df00224bcf304d3385f0099df24502954ab5" exitCode=0 Oct 30 18:45:08.085790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.085467 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" event={"ID":"d04094c2-6fe0-4ae1-a731-b8679713e79f","Type":"ContainerDied","Data":"0dec8ee1a98d1008c3e60100da21df00224bcf304d3385f0099df24502954ab5"} Oct 30 18:45:08.203900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.203873 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:45:08.256209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.256183 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-kube-api-access-9fd5k\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-kube-api-access-9fd5k\") pod \"d04094c2-6fe0-4ae1-a731-b8679713e79f\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " Oct 30 18:45:08.256402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.256301 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials\") pod \"d04094c2-6fe0-4ae1-a731-b8679713e79f\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " Oct 30 18:45:08.256607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.256580 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-plugins-config-volume\") pod \"d04094c2-6fe0-4ae1-a731-b8679713e79f\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " Oct 30 18:45:08.256730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.256674 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-routing-overlay-volume\") pod \"d04094c2-6fe0-4ae1-a731-b8679713e79f\" (UID: \"d04094c2-6fe0-4ae1-a731-b8679713e79f\") " Oct 30 18:45:08.257128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.257010 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-plugins-config-volume" pod "d04094c2-6fe0-4ae1-a731-b8679713e79f" (UID: "d04094c2-6fe0-4ae1-a731-b8679713e79f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:08.257228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.257191 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-routing-overlay-volume" pod "d04094c2-6fe0-4ae1-a731-b8679713e79f" (UID: "d04094c2-6fe0-4ae1-a731-b8679713e79f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:08.257627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.257606 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:08.257721 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.257632 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d04094c2-6fe0-4ae1-a731-b8679713e79f-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:08.258959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.258935 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-kube-api-access-9fd5k" pod "d04094c2-6fe0-4ae1-a731-b8679713e79f" (UID: "d04094c2-6fe0-4ae1-a731-b8679713e79f"). InnerVolumeSpecName "kube-api-access-9fd5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:08.259292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.259231 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials" pod "d04094c2-6fe0-4ae1-a731-b8679713e79f" (UID: "d04094c2-6fe0-4ae1-a731-b8679713e79f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:08.359508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.359479 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fd5k\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-kube-api-access-9fd5k\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:08.359508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.359505 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d04094c2-6fe0-4ae1-a731-b8679713e79f-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:08.764963 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:08.764934 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:08.765134 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:08.765095 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:08.765134 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:08.765118 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx: secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:08.765244 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:08.765170 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials podName:f5920a67-84ce-4ede-896e-2de205b0f63d nodeName:}" failed. No retries permitted until 2026-10-30 18:45:16.765156127 +0000 UTC m=+2815.814201038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" (UID: "f5920a67-84ce-4ede-896e-2de205b0f63d") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:45:09.094639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:09.094612 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" Oct 30 18:45:09.095017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:09.094651 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh" event={"ID":"d04094c2-6fe0-4ae1-a731-b8679713e79f","Type":"ContainerDied","Data":"2341c4920735648eb4a1491871ba038d61ddb49d49f228564736b88a32dd4aa3"} Oct 30 18:45:09.095017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:09.094720 4862 scope.go:122] "RemoveContainer" containerID="0dec8ee1a98d1008c3e60100da21df00224bcf304d3385f0099df24502954ab5" Oct 30 18:45:09.146816 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:09.146795 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh"] Oct 30 18:45:09.158826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:09.158805 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cdh8qh"] Oct 30 18:45:11.064520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:11.064488 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d04094c2-6fe0-4ae1-a731-b8679713e79f" path="/var/lib/kubelet/pods/d04094c2-6fe0-4ae1-a731-b8679713e79f/volumes" Oct 30 18:45:13.959238 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:13.959170 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:45:14.032209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.032176 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx"] Oct 30 18:45:14.032746 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:14.032716 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" podUID="f5920a67-84ce-4ede-896e-2de205b0f63d" Oct 30 18:45:14.133359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.133332 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:14.142832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.142808 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:14.229302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.229210 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-plugins-config-volume\") pod \"f5920a67-84ce-4ede-896e-2de205b0f63d\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " Oct 30 18:45:14.229302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.229247 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-routing-overlay-volume\") pod \"f5920a67-84ce-4ede-896e-2de205b0f63d\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " Oct 30 18:45:14.229511 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.229362 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-kube-api-access-w5f7d\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-kube-api-access-w5f7d\") pod \"f5920a67-84ce-4ede-896e-2de205b0f63d\" (UID: \"f5920a67-84ce-4ede-896e-2de205b0f63d\") " Oct 30 18:45:14.229636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.229612 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-plugins-config-volume" pod "f5920a67-84ce-4ede-896e-2de205b0f63d" (UID: "f5920a67-84ce-4ede-896e-2de205b0f63d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:14.229694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.229654 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-routing-overlay-volume" pod "f5920a67-84ce-4ede-896e-2de205b0f63d" (UID: "f5920a67-84ce-4ede-896e-2de205b0f63d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:14.229858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.229843 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:14.229897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.229861 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5920a67-84ce-4ede-896e-2de205b0f63d-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:14.231542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.231518 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-kube-api-access-w5f7d" pod "f5920a67-84ce-4ede-896e-2de205b0f63d" (UID: "f5920a67-84ce-4ede-896e-2de205b0f63d"). InnerVolumeSpecName "kube-api-access-w5f7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:14.331339 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:14.331317 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5f7d\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-kube-api-access-w5f7d\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:15.140488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:15.140464 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx" Oct 30 18:45:15.222087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:15.222053 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx"] Oct 30 18:45:15.227327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:15.227296 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-555ff7kbx"] Oct 30 18:45:15.341450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:15.341388 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5920a67-84ce-4ede-896e-2de205b0f63d-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:16.055342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:16.055306 4862 scope.go:122] "RemoveContainer" containerID="3c92dfe2bd5e206e39170470662e77300916138d1c95ad20d0804f6d47414ce3" Oct 30 18:45:16.055508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:16.055435 4862 scope.go:122] "RemoveContainer" containerID="0113968fcbe0fdbe9bd293bb7bef053fd6195d086f24f8ba05a1d3a7d9c77c17" Oct 30 18:45:16.055714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:16.055694 4862 scope.go:122] "RemoveContainer" containerID="814af0f682351c6f0624c61703cea6234d7eb9da1f8396f677339d2cc1a7a5f2" Oct 30 18:45:17.063559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:17.063533 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5920a67-84ce-4ede-896e-2de205b0f63d" path="/var/lib/kubelet/pods/f5920a67-84ce-4ede-896e-2de205b0f63d/volumes" Oct 30 18:45:17.064244 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:17.064226 4862 scope.go:122] "RemoveContainer" containerID="18cedf1c89f5dba79dae9e100936e3587a52fe6481ce3d112e2f1605c490cb7e" Oct 30 18:45:17.064424 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:17.064406 4862 scope.go:122] "RemoveContainer" containerID="06beb9bb9d0d2d0f6b229e4994f2f0f0a5cc89fa60f0b03fd7b1ab287f2828c7" Oct 30 18:45:17.157948 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:17.157923 4862 generic.go:356] "Generic (PLEG): container finished" podID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerID="f52cf8127d687f6dfdd39b1084aaadd4ef1c05bed9f70f8a68bb1970e0a2f313" exitCode=60 Oct 30 18:45:17.158074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:17.157993 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" event={"ID":"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca","Type":"ContainerDied","Data":"f52cf8127d687f6dfdd39b1084aaadd4ef1c05bed9f70f8a68bb1970e0a2f313"} Oct 30 18:45:17.158074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:17.158053 4862 scope.go:122] "RemoveContainer" containerID="814af0f682351c6f0624c61703cea6234d7eb9da1f8396f677339d2cc1a7a5f2" Oct 30 18:45:17.158466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:17.158448 4862 scope.go:122] "RemoveContainer" containerID="f52cf8127d687f6dfdd39b1084aaadd4ef1c05bed9f70f8a68bb1970e0a2f313" Oct 30 18:45:17.158779 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:17.158759 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc_odh-ai-gateway-infra(b50bc46b-93f1-40f9-a6e3-78bf998aa2ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" Oct 30 18:45:17.160006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:17.159986 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" event={"ID":"bb2fed97-f53d-4f11-a967-8b975837a13e","Type":"ContainerStarted","Data":"6e9a5ab98bbd827ca88968f0ee31ada67438d8f8cfebb841b9487ff1111bb7fb"} Oct 30 18:45:17.161884 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:17.161862 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" event={"ID":"0c951a41-65ad-48dd-bb7e-1cfde1ef853f","Type":"ContainerStarted","Data":"707806767b4282e51f92c6eaaa177f0eb0af652c0c730cc3445ad0acad05e8ce"} Oct 30 18:45:18.055052 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.055027 4862 scope.go:122] "RemoveContainer" containerID="b7239eb43ffd66ddadd16da758b244cb780a9a25b3eaf76decd88659a175c4ca" Oct 30 18:45:18.055201 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.055150 4862 scope.go:122] "RemoveContainer" containerID="32ef75be542dc2c2e329262fc6d86c7c5225cf892883120dad9cb1af98217abd" Oct 30 18:45:18.055317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.055298 4862 scope.go:122] "RemoveContainer" containerID="74210ddc56ddad30bcb6f6622a43233c58726a94f1fdbbfeb0b9db435689ccc7" Oct 30 18:45:18.171319 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.171289 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" event={"ID":"dc189cb4-8886-479c-903a-752c5ad19901","Type":"ContainerStarted","Data":"8026c206fa5645811e810442a74fcb90b2c2307670db7a20e7876770306a15c1"} Oct 30 18:45:18.174806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.174784 4862 generic.go:356] "Generic (PLEG): container finished" podID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerID="6e9a5ab98bbd827ca88968f0ee31ada67438d8f8cfebb841b9487ff1111bb7fb" exitCode=60 Oct 30 18:45:18.174918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.174854 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" event={"ID":"bb2fed97-f53d-4f11-a967-8b975837a13e","Type":"ContainerDied","Data":"6e9a5ab98bbd827ca88968f0ee31ada67438d8f8cfebb841b9487ff1111bb7fb"} Oct 30 18:45:18.174918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.174902 4862 scope.go:122] "RemoveContainer" containerID="0113968fcbe0fdbe9bd293bb7bef053fd6195d086f24f8ba05a1d3a7d9c77c17" Oct 30 18:45:18.175222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.175200 4862 scope.go:122] "RemoveContainer" containerID="6e9a5ab98bbd827ca88968f0ee31ada67438d8f8cfebb841b9487ff1111bb7fb" Oct 30 18:45:18.175491 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:18.175471 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-dtlxv_odh-ai-gateway-infra(bb2fed97-f53d-4f11-a967-8b975837a13e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" Oct 30 18:45:18.177170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.177147 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" event={"ID":"bb2168db-5894-4b73-ba78-754f1d8863bc","Type":"ContainerStarted","Data":"1343c46b750e2d926726043a8c5a34befb82fb00ec1402f094d0a564e92b4c27"} Oct 30 18:45:18.178904 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.178884 4862 generic.go:356] "Generic (PLEG): container finished" podID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerID="707806767b4282e51f92c6eaaa177f0eb0af652c0c730cc3445ad0acad05e8ce" exitCode=60 Oct 30 18:45:18.178998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.178946 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" event={"ID":"0c951a41-65ad-48dd-bb7e-1cfde1ef853f","Type":"ContainerDied","Data":"707806767b4282e51f92c6eaaa177f0eb0af652c0c730cc3445ad0acad05e8ce"} Oct 30 18:45:18.179228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.179215 4862 scope.go:122] "RemoveContainer" containerID="707806767b4282e51f92c6eaaa177f0eb0af652c0c730cc3445ad0acad05e8ce" Oct 30 18:45:18.179468 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:18.179453 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn_odh-ai-gateway-infra(0c951a41-65ad-48dd-bb7e-1cfde1ef853f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" Oct 30 18:45:18.190003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.189984 4862 scope.go:122] "RemoveContainer" containerID="3c92dfe2bd5e206e39170470662e77300916138d1c95ad20d0804f6d47414ce3" Oct 30 18:45:18.212984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.212967 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc"] Oct 30 18:45:18.260067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.260041 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn"] Oct 30 18:45:18.323152 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.323080 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv"] Oct 30 18:45:18.402138 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.402117 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:18.501814 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.501789 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-maas-api-service-ca\") pod \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\" (UID: \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\") " Oct 30 18:45:18.502016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.501997 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-kube-api-access-pc766\" (UniqueName: \"kubernetes.io/projected/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-kube-api-access-pc766\") pod \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\" (UID: \"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca\") " Oct 30 18:45:18.502184 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.502163 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-maas-api-service-ca" pod "b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" (UID: "b50bc46b-93f1-40f9-a6e3-78bf998aa2ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:18.502526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.502510 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:18.530408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.530383 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-kube-api-access-pc766" pod "b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" (UID: "b50bc46b-93f1-40f9-a6e3-78bf998aa2ca"). InnerVolumeSpecName "kube-api-access-pc766". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:18.603591 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:18.603531 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc766\" (UniqueName: \"kubernetes.io/projected/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca-kube-api-access-pc766\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:19.068148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.068120 4862 scope.go:122] "RemoveContainer" containerID="520f430d301a9b4a4793db8f6234b7a25f2acca92a2819fd6db4a8b94f7272c8" Oct 30 18:45:19.068376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.068354 4862 scope.go:122] "RemoveContainer" containerID="182d755e882bc359c49288be1a6fde2a835d0027a2f3e9742b33d882879a6c3b" Oct 30 18:45:19.068633 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.068597 4862 scope.go:122] "RemoveContainer" containerID="1dc1cb2f01a15be7d2475051a0722bef96900d7d3c42884c95e22ce5bf21639e" Oct 30 18:45:19.068760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.068698 4862 scope.go:122] "RemoveContainer" containerID="9dd2808e445e7ec566a48a20fce7d1a5137034aa6efa14748e8c9e7659628835" Oct 30 18:45:19.068916 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.068809 4862 scope.go:122] "RemoveContainer" containerID="77d195cfaffc0e3faa922a4c7d70ad05110ef61681302b62dbf76070a127b6f0" Oct 30 18:45:19.187155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.187129 4862 generic.go:356] "Generic (PLEG): container finished" podID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerID="1343c46b750e2d926726043a8c5a34befb82fb00ec1402f094d0a564e92b4c27" exitCode=60 Oct 30 18:45:19.187588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.187197 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" event={"ID":"bb2168db-5894-4b73-ba78-754f1d8863bc","Type":"ContainerDied","Data":"1343c46b750e2d926726043a8c5a34befb82fb00ec1402f094d0a564e92b4c27"} Oct 30 18:45:19.187588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.187252 4862 scope.go:122] "RemoveContainer" containerID="06beb9bb9d0d2d0f6b229e4994f2f0f0a5cc89fa60f0b03fd7b1ab287f2828c7" Oct 30 18:45:19.187702 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.187641 4862 scope.go:122] "RemoveContainer" containerID="1343c46b750e2d926726043a8c5a34befb82fb00ec1402f094d0a564e92b4c27" Oct 30 18:45:19.187995 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:19.187974 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf_odh-ai-gateway-infra(bb2168db-5894-4b73-ba78-754f1d8863bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" Oct 30 18:45:19.191668 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.191646 4862 generic.go:356] "Generic (PLEG): container finished" podID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerID="cad0b68964c1678a41c81506ca52ab1734e015ed2af2b35343af476af9b5fc1a" exitCode=60 Oct 30 18:45:19.191759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.191726 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" event={"ID":"06cf935f-86dc-4790-93b2-c7baea9b6a42","Type":"ContainerDied","Data":"cad0b68964c1678a41c81506ca52ab1734e015ed2af2b35343af476af9b5fc1a"} Oct 30 18:45:19.192896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.192875 4862 scope.go:122] "RemoveContainer" containerID="cad0b68964c1678a41c81506ca52ab1734e015ed2af2b35343af476af9b5fc1a" Oct 30 18:45:19.193348 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:19.193308 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww_odh-ai-gateway-infra(06cf935f-86dc-4790-93b2-c7baea9b6a42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" Oct 30 18:45:19.194417 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.194223 4862 generic.go:356] "Generic (PLEG): container finished" podID="cfffb210-225d-4707-9bdd-4608be2dc749" containerID="efe7f3a92ed5578f5bf0d4a263f3a1ed10ff9f2389c40a3b592d0bec41cb6942" exitCode=60 Oct 30 18:45:19.194417 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.194309 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" event={"ID":"cfffb210-225d-4707-9bdd-4608be2dc749","Type":"ContainerDied","Data":"efe7f3a92ed5578f5bf0d4a263f3a1ed10ff9f2389c40a3b592d0bec41cb6942"} Oct 30 18:45:19.195227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.194747 4862 scope.go:122] "RemoveContainer" containerID="efe7f3a92ed5578f5bf0d4a263f3a1ed10ff9f2389c40a3b592d0bec41cb6942" Oct 30 18:45:19.195227 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:19.195183 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm_odh-ai-gateway-infra(cfffb210-225d-4707-9bdd-4608be2dc749)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" Oct 30 18:45:19.197393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.197373 4862 generic.go:356] "Generic (PLEG): container finished" podID="dc189cb4-8886-479c-903a-752c5ad19901" containerID="8026c206fa5645811e810442a74fcb90b2c2307670db7a20e7876770306a15c1" exitCode=60 Oct 30 18:45:19.197506 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.197483 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" event={"ID":"dc189cb4-8886-479c-903a-752c5ad19901","Type":"ContainerDied","Data":"8026c206fa5645811e810442a74fcb90b2c2307670db7a20e7876770306a15c1"} Oct 30 18:45:19.197809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.197794 4862 scope.go:122] "RemoveContainer" containerID="8026c206fa5645811e810442a74fcb90b2c2307670db7a20e7876770306a15c1" Oct 30 18:45:19.198075 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:19.198046 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg_odh-ai-gateway-infra(dc189cb4-8886-479c-903a-752c5ad19901)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" podUID="dc189cb4-8886-479c-903a-752c5ad19901" Oct 30 18:45:19.199223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.199191 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" Oct 30 18:45:19.199351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.199202 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc" event={"ID":"b50bc46b-93f1-40f9-a6e3-78bf998aa2ca","Type":"ContainerDied","Data":"ac05f999df790f60d350f6323755984962300a54efde3376a14ac73dd5377022"} Oct 30 18:45:19.203960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.203907 4862 generic.go:356] "Generic (PLEG): container finished" podID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerID="cb5816fc5fa1bf2d2c592f406b8f9eb9f8782c4df8eeddfc96b9a187c13e3b9a" exitCode=60 Oct 30 18:45:19.203960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.203930 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" event={"ID":"edb49158-4950-4fcb-9ddc-aaa5005a019f","Type":"ContainerDied","Data":"cb5816fc5fa1bf2d2c592f406b8f9eb9f8782c4df8eeddfc96b9a187c13e3b9a"} Oct 30 18:45:19.206360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.204321 4862 scope.go:122] "RemoveContainer" containerID="cb5816fc5fa1bf2d2c592f406b8f9eb9f8782c4df8eeddfc96b9a187c13e3b9a" Oct 30 18:45:19.206360 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:19.204572 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z_odh-ai-gateway-infra(edb49158-4950-4fcb-9ddc-aaa5005a019f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" Oct 30 18:45:19.240537 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.240503 4862 scope.go:122] "RemoveContainer" containerID="32ef75be542dc2c2e329262fc6d86c7c5225cf892883120dad9cb1af98217abd" Oct 30 18:45:19.241868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.241844 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg"] Oct 30 18:45:19.264147 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.264013 4862 scope.go:122] "RemoveContainer" containerID="b7239eb43ffd66ddadd16da758b244cb780a9a25b3eaf76decd88659a175c4ca" Oct 30 18:45:19.270652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.270630 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf"] Oct 30 18:45:19.293597 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.293573 4862 scope.go:122] "RemoveContainer" containerID="18cedf1c89f5dba79dae9e100936e3587a52fe6481ce3d112e2f1605c490cb7e" Oct 30 18:45:19.313236 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.313167 4862 scope.go:122] "RemoveContainer" containerID="f52cf8127d687f6dfdd39b1084aaadd4ef1c05bed9f70f8a68bb1970e0a2f313" Oct 30 18:45:19.328804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.328703 4862 scope.go:122] "RemoveContainer" containerID="74210ddc56ddad30bcb6f6622a43233c58726a94f1fdbbfeb0b9db435689ccc7" Oct 30 18:45:19.457623 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.457595 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:19.460867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.460839 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:19.621379 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.621350 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc"] Oct 30 18:45:19.621742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.621714 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-maas-api-service-ca\") pod \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\" (UID: \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\") " Oct 30 18:45:19.621893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.621755 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2fed97-f53d-4f11-a967-8b975837a13e-kube-api-access-cg6m9\" (UniqueName: \"kubernetes.io/projected/bb2fed97-f53d-4f11-a967-8b975837a13e-kube-api-access-cg6m9\") pod \"bb2fed97-f53d-4f11-a967-8b975837a13e\" (UID: \"bb2fed97-f53d-4f11-a967-8b975837a13e\") " Oct 30 18:45:19.621893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.621787 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-kube-api-access-4sw4x\" (UniqueName: \"kubernetes.io/projected/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-kube-api-access-4sw4x\") pod \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\" (UID: \"0c951a41-65ad-48dd-bb7e-1cfde1ef853f\") " Oct 30 18:45:19.622022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.621899 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2fed97-f53d-4f11-a967-8b975837a13e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2fed97-f53d-4f11-a967-8b975837a13e-maas-api-service-ca\") pod \"bb2fed97-f53d-4f11-a967-8b975837a13e\" (UID: \"bb2fed97-f53d-4f11-a967-8b975837a13e\") " Oct 30 18:45:19.622431 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.622407 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-maas-api-service-ca" pod "0c951a41-65ad-48dd-bb7e-1cfde1ef853f" (UID: "0c951a41-65ad-48dd-bb7e-1cfde1ef853f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:19.622526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.622433 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2fed97-f53d-4f11-a967-8b975837a13e-maas-api-service-ca" pod "bb2fed97-f53d-4f11-a967-8b975837a13e" (UID: "bb2fed97-f53d-4f11-a967-8b975837a13e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:19.624726 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.624705 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2fed97-f53d-4f11-a967-8b975837a13e-kube-api-access-cg6m9" pod "bb2fed97-f53d-4f11-a967-8b975837a13e" (UID: "bb2fed97-f53d-4f11-a967-8b975837a13e"). InnerVolumeSpecName "kube-api-access-cg6m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:19.625847 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.625831 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-kube-api-access-4sw4x" pod "0c951a41-65ad-48dd-bb7e-1cfde1ef853f" (UID: "0c951a41-65ad-48dd-bb7e-1cfde1ef853f"). InnerVolumeSpecName "kube-api-access-4sw4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:19.627839 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.627816 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889765-kxxmc"] Oct 30 18:45:19.723250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.723217 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2fed97-f53d-4f11-a967-8b975837a13e-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:19.723250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.723247 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:19.723250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.723276 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6m9\" (UniqueName: \"kubernetes.io/projected/bb2fed97-f53d-4f11-a967-8b975837a13e-kube-api-access-cg6m9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:19.723510 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:19.723288 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sw4x\" (UniqueName: \"kubernetes.io/projected/0c951a41-65ad-48dd-bb7e-1cfde1ef853f-kube-api-access-4sw4x\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:19.877798 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:19.877722 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:45:20.222081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.222045 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" event={"ID":"e4d5532c-91de-471d-bdd7-e9ad1b0ce399","Type":"ContainerStarted","Data":"cf29d31c5704568cf0a0b2fad55943e9c837b4dc3aa7a147439e24c34374fc02"} Oct 30 18:45:20.224240 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.224209 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" event={"ID":"a97dc6f5-10b4-4d60-90bf-423a8da7ab74","Type":"ContainerStarted","Data":"2d79ddb7d27ef67efc412bfc5556c8487753f5643f27b4beb451aa6a22157f84"} Oct 30 18:45:20.226106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.226072 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" event={"ID":"dbf488c3-95ac-4149-9b8b-ebb4522cc909","Type":"ContainerStarted","Data":"48e241ad7ce4156bc31beb48f3d76f15af286230b90ea6dcf59ed901bc5c8fa6"} Oct 30 18:45:20.229531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.229507 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" event={"ID":"bb2fed97-f53d-4f11-a967-8b975837a13e","Type":"ContainerDied","Data":"c6da7f1ead1d8103617c0e12794a26f27a46a094e4ce80d52c246433ee8b133a"} Oct 30 18:45:20.229729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.229706 4862 scope.go:122] "RemoveContainer" containerID="6e9a5ab98bbd827ca88968f0ee31ada67438d8f8cfebb841b9487ff1111bb7fb" Oct 30 18:45:20.230308 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.229856 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv" Oct 30 18:45:20.235164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.235134 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" event={"ID":"1ac031fb-b03c-4a94-9623-e19a1f4296e5","Type":"ContainerStarted","Data":"16ddefd5a24aff31f735bacfdc7dbda5b4860e6438840c9869dc1218c9650611"} Oct 30 18:45:20.243253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.242532 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" event={"ID":"8c1bb6bd-ca03-45c2-b563-0f884658acd0","Type":"ContainerStarted","Data":"240252f2cb0ad0ce635b9829f1475ece0f3bb891bfe5ac473632ab34700d9085"} Oct 30 18:45:20.244718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.244682 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" event={"ID":"0c951a41-65ad-48dd-bb7e-1cfde1ef853f","Type":"ContainerDied","Data":"39da8ea41ac306c48dd1c06c4d5eaf75aecc7cf58c76340cb7ac0c4eb2c37980"} Oct 30 18:45:20.244808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.244797 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn" Oct 30 18:45:20.291969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.291927 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww"] Oct 30 18:45:20.388556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.388162 4862 scope.go:122] "RemoveContainer" containerID="707806767b4282e51f92c6eaaa177f0eb0af652c0c730cc3445ad0acad05e8ce" Oct 30 18:45:20.392514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.392327 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm"] Oct 30 18:45:20.512977 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.512950 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:20.521404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.521328 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:20.522718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.522505 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z"] Oct 30 18:45:20.642342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.635147 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2168db-5894-4b73-ba78-754f1d8863bc-kube-api-access-x68kt\" (UniqueName: \"kubernetes.io/projected/bb2168db-5894-4b73-ba78-754f1d8863bc-kube-api-access-x68kt\") pod \"bb2168db-5894-4b73-ba78-754f1d8863bc\" (UID: \"bb2168db-5894-4b73-ba78-754f1d8863bc\") " Oct 30 18:45:20.642342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.635280 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc189cb4-8886-479c-903a-752c5ad19901-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc189cb4-8886-479c-903a-752c5ad19901-maas-api-service-ca\") pod \"dc189cb4-8886-479c-903a-752c5ad19901\" (UID: \"dc189cb4-8886-479c-903a-752c5ad19901\") " Oct 30 18:45:20.642342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.635351 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc189cb4-8886-479c-903a-752c5ad19901-kube-api-access-m2v5k\" (UniqueName: \"kubernetes.io/projected/dc189cb4-8886-479c-903a-752c5ad19901-kube-api-access-m2v5k\") pod \"dc189cb4-8886-479c-903a-752c5ad19901\" (UID: \"dc189cb4-8886-479c-903a-752c5ad19901\") " Oct 30 18:45:20.642342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.635448 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2168db-5894-4b73-ba78-754f1d8863bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2168db-5894-4b73-ba78-754f1d8863bc-maas-api-service-ca\") pod \"bb2168db-5894-4b73-ba78-754f1d8863bc\" (UID: \"bb2168db-5894-4b73-ba78-754f1d8863bc\") " Oct 30 18:45:20.642342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.636327 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2168db-5894-4b73-ba78-754f1d8863bc-maas-api-service-ca" pod "bb2168db-5894-4b73-ba78-754f1d8863bc" (UID: "bb2168db-5894-4b73-ba78-754f1d8863bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:20.642342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.637025 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc189cb4-8886-479c-903a-752c5ad19901-maas-api-service-ca" pod "dc189cb4-8886-479c-903a-752c5ad19901" (UID: "dc189cb4-8886-479c-903a-752c5ad19901"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:20.642822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.642801 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc189cb4-8886-479c-903a-752c5ad19901-kube-api-access-m2v5k" pod "dc189cb4-8886-479c-903a-752c5ad19901" (UID: "dc189cb4-8886-479c-903a-752c5ad19901"). InnerVolumeSpecName "kube-api-access-m2v5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:20.644483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.644237 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:20.647762 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.647727 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:20.652888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.652862 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2168db-5894-4b73-ba78-754f1d8863bc-kube-api-access-x68kt" pod "bb2168db-5894-4b73-ba78-754f1d8863bc" (UID: "bb2168db-5894-4b73-ba78-754f1d8863bc"). InnerVolumeSpecName "kube-api-access-x68kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:20.682867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.682356 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv"] Oct 30 18:45:20.688780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.688757 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-dtlxv"] Oct 30 18:45:20.695939 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.695919 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:20.737069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.737010 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfffb210-225d-4707-9bdd-4608be2dc749-kube-api-access-8k75g\" (UniqueName: \"kubernetes.io/projected/cfffb210-225d-4707-9bdd-4608be2dc749-kube-api-access-8k75g\") pod \"cfffb210-225d-4707-9bdd-4608be2dc749\" (UID: \"cfffb210-225d-4707-9bdd-4608be2dc749\") " Oct 30 18:45:20.737185 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.737139 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfffb210-225d-4707-9bdd-4608be2dc749-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfffb210-225d-4707-9bdd-4608be2dc749-maas-api-service-ca\") pod \"cfffb210-225d-4707-9bdd-4608be2dc749\" (UID: \"cfffb210-225d-4707-9bdd-4608be2dc749\") " Oct 30 18:45:20.737278 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.737193 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06cf935f-86dc-4790-93b2-c7baea9b6a42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cf935f-86dc-4790-93b2-c7baea9b6a42-maas-api-service-ca\") pod \"06cf935f-86dc-4790-93b2-c7baea9b6a42\" (UID: \"06cf935f-86dc-4790-93b2-c7baea9b6a42\") " Oct 30 18:45:20.737336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.737250 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06cf935f-86dc-4790-93b2-c7baea9b6a42-kube-api-access-dcjpt\" (UniqueName: \"kubernetes.io/projected/06cf935f-86dc-4790-93b2-c7baea9b6a42-kube-api-access-dcjpt\") pod \"06cf935f-86dc-4790-93b2-c7baea9b6a42\" (UID: \"06cf935f-86dc-4790-93b2-c7baea9b6a42\") " Oct 30 18:45:20.737535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.737510 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfffb210-225d-4707-9bdd-4608be2dc749-maas-api-service-ca" pod "cfffb210-225d-4707-9bdd-4608be2dc749" (UID: "cfffb210-225d-4707-9bdd-4608be2dc749"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:20.737628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.737569 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06cf935f-86dc-4790-93b2-c7baea9b6a42-maas-api-service-ca" pod "06cf935f-86dc-4790-93b2-c7baea9b6a42" (UID: "06cf935f-86dc-4790-93b2-c7baea9b6a42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:20.738105 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.738084 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc189cb4-8886-479c-903a-752c5ad19901-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.738146 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.738113 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2v5k\" (UniqueName: \"kubernetes.io/projected/dc189cb4-8886-479c-903a-752c5ad19901-kube-api-access-m2v5k\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.738146 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.738130 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfffb210-225d-4707-9bdd-4608be2dc749-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.738234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.738145 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2168db-5894-4b73-ba78-754f1d8863bc-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.738234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.738159 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cf935f-86dc-4790-93b2-c7baea9b6a42-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.738234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.738173 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68kt\" (UniqueName: \"kubernetes.io/projected/bb2168db-5894-4b73-ba78-754f1d8863bc-kube-api-access-x68kt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.739489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.739468 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfffb210-225d-4707-9bdd-4608be2dc749-kube-api-access-8k75g" pod "cfffb210-225d-4707-9bdd-4608be2dc749" (UID: "cfffb210-225d-4707-9bdd-4608be2dc749"). InnerVolumeSpecName "kube-api-access-8k75g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:20.739568 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.739518 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cf935f-86dc-4790-93b2-c7baea9b6a42-kube-api-access-dcjpt" pod "06cf935f-86dc-4790-93b2-c7baea9b6a42" (UID: "06cf935f-86dc-4790-93b2-c7baea9b6a42"). InnerVolumeSpecName "kube-api-access-dcjpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:20.839579 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.839545 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb49158-4950-4fcb-9ddc-aaa5005a019f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb49158-4950-4fcb-9ddc-aaa5005a019f-maas-api-service-ca\") pod \"edb49158-4950-4fcb-9ddc-aaa5005a019f\" (UID: \"edb49158-4950-4fcb-9ddc-aaa5005a019f\") " Oct 30 18:45:20.839760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.839660 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb49158-4950-4fcb-9ddc-aaa5005a019f-kube-api-access-544dr\" (UniqueName: \"kubernetes.io/projected/edb49158-4950-4fcb-9ddc-aaa5005a019f-kube-api-access-544dr\") pod \"edb49158-4950-4fcb-9ddc-aaa5005a019f\" (UID: \"edb49158-4950-4fcb-9ddc-aaa5005a019f\") " Oct 30 18:45:20.839961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.839933 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb49158-4950-4fcb-9ddc-aaa5005a019f-maas-api-service-ca" pod "edb49158-4950-4fcb-9ddc-aaa5005a019f" (UID: "edb49158-4950-4fcb-9ddc-aaa5005a019f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:20.840371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.840340 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edb49158-4950-4fcb-9ddc-aaa5005a019f-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.840371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.840368 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k75g\" (UniqueName: \"kubernetes.io/projected/cfffb210-225d-4707-9bdd-4608be2dc749-kube-api-access-8k75g\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.840520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.840383 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcjpt\" (UniqueName: \"kubernetes.io/projected/06cf935f-86dc-4790-93b2-c7baea9b6a42-kube-api-access-dcjpt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.842901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.842874 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb49158-4950-4fcb-9ddc-aaa5005a019f-kube-api-access-544dr" pod "edb49158-4950-4fcb-9ddc-aaa5005a019f" (UID: "edb49158-4950-4fcb-9ddc-aaa5005a019f"). InnerVolumeSpecName "kube-api-access-544dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:20.873287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.861092 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn"] Oct 30 18:45:20.873287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.872375 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889765-zlxdn"] Oct 30 18:45:20.942763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:20.942734 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-544dr\" (UniqueName: \"kubernetes.io/projected/edb49158-4950-4fcb-9ddc-aaa5005a019f-kube-api-access-544dr\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:21.069686 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.069627 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" path="/var/lib/kubelet/pods/0c951a41-65ad-48dd-bb7e-1cfde1ef853f/volumes" Oct 30 18:45:21.086483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.080959 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" path="/var/lib/kubelet/pods/b50bc46b-93f1-40f9-a6e3-78bf998aa2ca/volumes" Oct 30 18:45:21.120291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.095989 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" path="/var/lib/kubelet/pods/bb2fed97-f53d-4f11-a967-8b975837a13e/volumes" Oct 30 18:45:21.309741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.299702 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" event={"ID":"dc189cb4-8886-479c-903a-752c5ad19901","Type":"ContainerDied","Data":"d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef"} Oct 30 18:45:21.309741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.300078 4862 scope.go:122] "RemoveContainer" containerID="8026c206fa5645811e810442a74fcb90b2c2307670db7a20e7876770306a15c1" Oct 30 18:45:21.309741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.300290 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg" Oct 30 18:45:21.316947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.316915 4862 generic.go:356] "Generic (PLEG): container finished" podID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerID="cf29d31c5704568cf0a0b2fad55943e9c837b4dc3aa7a147439e24c34374fc02" exitCode=60 Oct 30 18:45:21.317303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.317277 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" event={"ID":"e4d5532c-91de-471d-bdd7-e9ad1b0ce399","Type":"ContainerDied","Data":"cf29d31c5704568cf0a0b2fad55943e9c837b4dc3aa7a147439e24c34374fc02"} Oct 30 18:45:21.317975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.317942 4862 scope.go:122] "RemoveContainer" containerID="cf29d31c5704568cf0a0b2fad55943e9c837b4dc3aa7a147439e24c34374fc02" Oct 30 18:45:21.320309 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:21.318608 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv_odh-ai-gateway-infra(e4d5532c-91de-471d-bdd7-e9ad1b0ce399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" Oct 30 18:45:21.333728 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.332587 4862 generic.go:356] "Generic (PLEG): container finished" podID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerID="2d79ddb7d27ef67efc412bfc5556c8487753f5643f27b4beb451aa6a22157f84" exitCode=60 Oct 30 18:45:21.333728 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.332683 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" event={"ID":"a97dc6f5-10b4-4d60-90bf-423a8da7ab74","Type":"ContainerDied","Data":"2d79ddb7d27ef67efc412bfc5556c8487753f5643f27b4beb451aa6a22157f84"} Oct 30 18:45:21.333728 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.333116 4862 scope.go:122] "RemoveContainer" containerID="2d79ddb7d27ef67efc412bfc5556c8487753f5643f27b4beb451aa6a22157f84" Oct 30 18:45:21.333728 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.333384 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk"] Oct 30 18:45:21.333728 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:21.333510 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24_odh-ai-gateway-infra(a97dc6f5-10b4-4d60-90bf-423a8da7ab74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" Oct 30 18:45:21.337059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.337019 4862 generic.go:356] "Generic (PLEG): container finished" podID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerID="48e241ad7ce4156bc31beb48f3d76f15af286230b90ea6dcf59ed901bc5c8fa6" exitCode=60 Oct 30 18:45:21.337205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.337142 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" event={"ID":"dbf488c3-95ac-4149-9b8b-ebb4522cc909","Type":"ContainerDied","Data":"48e241ad7ce4156bc31beb48f3d76f15af286230b90ea6dcf59ed901bc5c8fa6"} Oct 30 18:45:21.343252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.343223 4862 generic.go:356] "Generic (PLEG): container finished" podID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerID="16ddefd5a24aff31f735bacfdc7dbda5b4860e6438840c9869dc1218c9650611" exitCode=60 Oct 30 18:45:21.344222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.343310 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" event={"ID":"1ac031fb-b03c-4a94-9623-e19a1f4296e5","Type":"ContainerDied","Data":"16ddefd5a24aff31f735bacfdc7dbda5b4860e6438840c9869dc1218c9650611"} Oct 30 18:45:21.346295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.344588 4862 scope.go:122] "RemoveContainer" containerID="16ddefd5a24aff31f735bacfdc7dbda5b4860e6438840c9869dc1218c9650611" Oct 30 18:45:21.346295 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:21.344919 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg_odh-ai-gateway-infra(1ac031fb-b03c-4a94-9623-e19a1f4296e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" Oct 30 18:45:21.351230 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.351195 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" Oct 30 18:45:21.351357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.351246 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z" event={"ID":"edb49158-4950-4fcb-9ddc-aaa5005a019f","Type":"ContainerDied","Data":"695c1689ba9e25855b4e2eade12650042396b2ca311f1a3b656108d37f22f574"} Oct 30 18:45:21.354131 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.354098 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" event={"ID":"bb2168db-5894-4b73-ba78-754f1d8863bc","Type":"ContainerDied","Data":"f7e5d04da28a047904a7d95f1b4847a4f46dcbb5d0c76287ae0dedb583d85feb"} Oct 30 18:45:21.354334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.354315 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf" Oct 30 18:45:21.354583 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.354567 4862 scope.go:122] "RemoveContainer" containerID="520f430d301a9b4a4793db8f6234b7a25f2acca92a2819fd6db4a8b94f7272c8" Oct 30 18:45:21.371611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.364688 4862 generic.go:356] "Generic (PLEG): container finished" podID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerID="240252f2cb0ad0ce635b9829f1475ece0f3bb891bfe5ac473632ab34700d9085" exitCode=60 Oct 30 18:45:21.371611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.364750 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" event={"ID":"8c1bb6bd-ca03-45c2-b563-0f884658acd0","Type":"ContainerDied","Data":"240252f2cb0ad0ce635b9829f1475ece0f3bb891bfe5ac473632ab34700d9085"} Oct 30 18:45:21.371611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.365031 4862 scope.go:122] "RemoveContainer" containerID="240252f2cb0ad0ce635b9829f1475ece0f3bb891bfe5ac473632ab34700d9085" Oct 30 18:45:21.371611 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:21.365246 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m_odh-ai-gateway-infra(8c1bb6bd-ca03-45c2-b563-0f884658acd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" Oct 30 18:45:21.371611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.370727 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg"] Oct 30 18:45:21.373007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.372329 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" event={"ID":"06cf935f-86dc-4790-93b2-c7baea9b6a42","Type":"ContainerDied","Data":"d86b2a47ad26d6522adaaf0033c5ad076fa965e50695d07941117542e75c7bcc"} Oct 30 18:45:21.373007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.372448 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww" Oct 30 18:45:21.375543 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.375522 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" event={"ID":"cfffb210-225d-4707-9bdd-4608be2dc749","Type":"ContainerDied","Data":"1e724f0a43da2cea67e5b95df1f5debaf643fd29a1ce1ec71ce5cdf51b5eb645"} Oct 30 18:45:21.375652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.375545 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm" Oct 30 18:45:21.381000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.380974 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m"] Oct 30 18:45:21.386161 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.386132 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889765-rmcdg"] Oct 30 18:45:21.389754 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.389736 4862 scope.go:122] "RemoveContainer" containerID="182d755e882bc359c49288be1a6fde2a835d0027a2f3e9742b33d882879a6c3b" Oct 30 18:45:21.411626 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.411564 4862 scope.go:122] "RemoveContainer" containerID="1dc1cb2f01a15be7d2475051a0722bef96900d7d3c42884c95e22ce5bf21639e" Oct 30 18:45:21.432092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.432074 4862 scope.go:122] "RemoveContainer" containerID="9dd2808e445e7ec566a48a20fce7d1a5137034aa6efa14748e8c9e7659628835" Oct 30 18:45:21.453249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.453227 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww"] Oct 30 18:45:21.460031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.459846 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e95c51-29889765-99gww"] Oct 30 18:45:21.511542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.507674 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg"] Oct 30 18:45:21.511542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.510385 4862 scope.go:122] "RemoveContainer" containerID="cb5816fc5fa1bf2d2c592f406b8f9eb9f8782c4df8eeddfc96b9a187c13e3b9a" Oct 30 18:45:21.582915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.578595 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24"] Oct 30 18:45:21.591349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.591307 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv"] Oct 30 18:45:21.639382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.639312 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf"] Oct 30 18:45:21.648318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.644552 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-38e0db-29889765-6qqdf"] Oct 30 18:45:21.662728 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.662634 4862 scope.go:122] "RemoveContainer" containerID="1343c46b750e2d926726043a8c5a34befb82fb00ec1402f094d0a564e92b4c27" Oct 30 18:45:21.666555 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.666529 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:21.679433 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.679408 4862 scope.go:122] "RemoveContainer" containerID="77d195cfaffc0e3faa922a4c7d70ad05110ef61681302b62dbf76070a127b6f0" Oct 30 18:45:21.697283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.694469 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm"] Oct 30 18:45:21.705281 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.702000 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889765-ddlbm"] Oct 30 18:45:21.715370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.715342 4862 scope.go:122] "RemoveContainer" containerID="cad0b68964c1678a41c81506ca52ab1734e015ed2af2b35343af476af9b5fc1a" Oct 30 18:45:21.733283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.729835 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z"] Oct 30 18:45:21.734463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.733976 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4a7508-29889765-d9s6z"] Oct 30 18:45:21.736387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.736345 4862 scope.go:122] "RemoveContainer" containerID="efe7f3a92ed5578f5bf0d4a263f3a1ed10ff9f2389c40a3b592d0bec41cb6942" Oct 30 18:45:21.783091 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.783059 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf488c3-95ac-4149-9b8b-ebb4522cc909-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf488c3-95ac-4149-9b8b-ebb4522cc909-maas-api-service-ca\") pod \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\" (UID: \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\") " Oct 30 18:45:21.783252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.783179 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf488c3-95ac-4149-9b8b-ebb4522cc909-kube-api-access-k9mz7\" (UniqueName: \"kubernetes.io/projected/dbf488c3-95ac-4149-9b8b-ebb4522cc909-kube-api-access-k9mz7\") pod \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\" (UID: \"dbf488c3-95ac-4149-9b8b-ebb4522cc909\") " Oct 30 18:45:21.784565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.784519 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf488c3-95ac-4149-9b8b-ebb4522cc909-maas-api-service-ca" pod "dbf488c3-95ac-4149-9b8b-ebb4522cc909" (UID: "dbf488c3-95ac-4149-9b8b-ebb4522cc909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:21.788462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.788438 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf488c3-95ac-4149-9b8b-ebb4522cc909-kube-api-access-k9mz7" pod "dbf488c3-95ac-4149-9b8b-ebb4522cc909" (UID: "dbf488c3-95ac-4149-9b8b-ebb4522cc909"). InnerVolumeSpecName "kube-api-access-k9mz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:21.885782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.885703 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf488c3-95ac-4149-9b8b-ebb4522cc909-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:21.885782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:21.885743 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9mz7\" (UniqueName: \"kubernetes.io/projected/dbf488c3-95ac-4149-9b8b-ebb4522cc909-kube-api-access-k9mz7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.190947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.190874 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst"] Oct 30 18:45:22.191505 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191491 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191506 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191521 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191526 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191536 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191541 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191549 4862 cpu_manager.go:414] "containerMap: removing container" podUID="dc189cb4-8886-479c-903a-752c5ad19901" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191553 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc189cb4-8886-479c-903a-752c5ad19901" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191561 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191565 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191570 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" containerName="cleanup" Oct 30 18:45:22.191569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191576 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191586 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d04094c2-6fe0-4ae1-a731-b8679713e79f" containerName="payload-processing" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191591 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04094c2-6fe0-4ae1-a731-b8679713e79f" containerName="payload-processing" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191598 4862 cpu_manager.go:414] "containerMap: removing container" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191602 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191608 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191613 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191619 4862 cpu_manager.go:414] "containerMap: removing container" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191623 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191631 4862 cpu_manager.go:414] "containerMap: removing container" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191635 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191645 4862 cpu_manager.go:414] "containerMap: removing container" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191652 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191666 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191673 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191685 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191692 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191703 4862 cpu_manager.go:414] "containerMap: removing container" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191710 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191726 4862 cpu_manager.go:414] "containerMap: removing container" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191730 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191736 4862 cpu_manager.go:414] "containerMap: removing container" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191741 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191754 4862 cpu_manager.go:414] "containerMap: removing container" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191758 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191765 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191769 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191775 4862 cpu_manager.go:414] "containerMap: removing container" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191779 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191790 4862 cpu_manager.go:414] "containerMap: removing container" podUID="dc189cb4-8886-479c-903a-752c5ad19901" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191794 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc189cb4-8886-479c-903a-752c5ad19901" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191802 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191807 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191817 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerName="cleanup" Oct 30 18:45:22.191912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191822 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191933 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191942 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191949 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191956 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191962 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc189cb4-8886-479c-903a-752c5ad19901" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191968 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191974 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191979 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191985 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191991 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.191996 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc189cb4-8886-479c-903a-752c5ad19901" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192001 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192006 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192013 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04094c2-6fe0-4ae1-a731-b8679713e79f" containerName="payload-processing" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192018 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192023 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192028 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192033 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192041 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192045 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192050 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerName="cleanup" Oct 30 18:45:22.192880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.192057 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerName="cleanup" Oct 30 18:45:22.197421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.197404 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.213055 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.213034 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6babca\"/\"agc-cred-abc040ceab35\"" Oct 30 18:45:22.214666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.214647 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst"] Oct 30 18:45:22.291621 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.291591 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.291764 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.291669 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.291821 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.291775 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stq8h\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-kube-api-access-stq8h\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.291854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.291836 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.387911 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.387876 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" Oct 30 18:45:22.388460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.387914 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk" event={"ID":"dbf488c3-95ac-4149-9b8b-ebb4522cc909","Type":"ContainerDied","Data":"a5cc852637e6081b4a7537edc3422dce921e900c8efef09be1697bc9cfa69136"} Oct 30 18:45:22.388460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.387989 4862 scope.go:122] "RemoveContainer" containerID="48e241ad7ce4156bc31beb48f3d76f15af286230b90ea6dcf59ed901bc5c8fa6" Oct 30 18:45:22.392686 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.392664 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.392824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.392742 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.392886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.392841 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stq8h\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-kube-api-access-stq8h\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.392939 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.392925 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.393577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.393548 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.394332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.394309 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.395934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.395909 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.437727 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.437698 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stq8h\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-kube-api-access-stq8h\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.511577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.511551 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:22.612372 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.608458 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:22.633458 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:22.633417 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache]" Oct 30 18:45:22.638526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.638418 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk"] Oct 30 18:45:22.645631 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.645276 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889765-zkgdk"] Oct 30 18:45:22.653239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.653213 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:22.698785 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.698727 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:22.701997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.701969 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-maas-api-service-ca\") pod \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\" (UID: \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\") " Oct 30 18:45:22.702087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.702051 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-kube-api-access-qkv7j\" (UniqueName: \"kubernetes.io/projected/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-kube-api-access-qkv7j\") pod \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\" (UID: \"a97dc6f5-10b4-4d60-90bf-423a8da7ab74\") " Oct 30 18:45:22.703280 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.703242 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-maas-api-service-ca" pod "a97dc6f5-10b4-4d60-90bf-423a8da7ab74" (UID: "a97dc6f5-10b4-4d60-90bf-423a8da7ab74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:22.705445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.705393 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-kube-api-access-qkv7j" pod "a97dc6f5-10b4-4d60-90bf-423a8da7ab74" (UID: "a97dc6f5-10b4-4d60-90bf-423a8da7ab74"). InnerVolumeSpecName "kube-api-access-qkv7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:22.715802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.715777 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:22.804030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.803994 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1bb6bd-ca03-45c2-b563-0f884658acd0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bb6bd-ca03-45c2-b563-0f884658acd0-maas-api-service-ca\") pod \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\" (UID: \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\") " Oct 30 18:45:22.804216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.804083 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac031fb-b03c-4a94-9623-e19a1f4296e5-kube-api-access-cclrt\" (UniqueName: \"kubernetes.io/projected/1ac031fb-b03c-4a94-9623-e19a1f4296e5-kube-api-access-cclrt\") pod \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\" (UID: \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\") " Oct 30 18:45:22.804216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.804171 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-maas-api-service-ca\") pod \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\" (UID: \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\") " Oct 30 18:45:22.804346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.804226 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-kube-api-access-72wfj\" (UniqueName: \"kubernetes.io/projected/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-kube-api-access-72wfj\") pod \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\" (UID: \"e4d5532c-91de-471d-bdd7-e9ad1b0ce399\") " Oct 30 18:45:22.804412 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.804356 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac031fb-b03c-4a94-9623-e19a1f4296e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac031fb-b03c-4a94-9623-e19a1f4296e5-maas-api-service-ca\") pod \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\" (UID: \"1ac031fb-b03c-4a94-9623-e19a1f4296e5\") " Oct 30 18:45:22.804467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.804429 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1bb6bd-ca03-45c2-b563-0f884658acd0-kube-api-access-djpfd\" (UniqueName: \"kubernetes.io/projected/8c1bb6bd-ca03-45c2-b563-0f884658acd0-kube-api-access-djpfd\") pod \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\" (UID: \"8c1bb6bd-ca03-45c2-b563-0f884658acd0\") " Oct 30 18:45:22.804967 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.804941 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.805098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.804970 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkv7j\" (UniqueName: \"kubernetes.io/projected/a97dc6f5-10b4-4d60-90bf-423a8da7ab74-kube-api-access-qkv7j\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.805673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.805649 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-maas-api-service-ca" pod "e4d5532c-91de-471d-bdd7-e9ad1b0ce399" (UID: "e4d5532c-91de-471d-bdd7-e9ad1b0ce399"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:22.806137 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.806116 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1bb6bd-ca03-45c2-b563-0f884658acd0-maas-api-service-ca" pod "8c1bb6bd-ca03-45c2-b563-0f884658acd0" (UID: "8c1bb6bd-ca03-45c2-b563-0f884658acd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:22.807503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.807482 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac031fb-b03c-4a94-9623-e19a1f4296e5-maas-api-service-ca" pod "1ac031fb-b03c-4a94-9623-e19a1f4296e5" (UID: "1ac031fb-b03c-4a94-9623-e19a1f4296e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:22.810442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.810092 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-kube-api-access-72wfj" pod "e4d5532c-91de-471d-bdd7-e9ad1b0ce399" (UID: "e4d5532c-91de-471d-bdd7-e9ad1b0ce399"). InnerVolumeSpecName "kube-api-access-72wfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:22.810442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.810161 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac031fb-b03c-4a94-9623-e19a1f4296e5-kube-api-access-cclrt" pod "1ac031fb-b03c-4a94-9623-e19a1f4296e5" (UID: "1ac031fb-b03c-4a94-9623-e19a1f4296e5"). InnerVolumeSpecName "kube-api-access-cclrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:22.819893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.819867 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1bb6bd-ca03-45c2-b563-0f884658acd0-kube-api-access-djpfd" pod "8c1bb6bd-ca03-45c2-b563-0f884658acd0" (UID: "8c1bb6bd-ca03-45c2-b563-0f884658acd0"). InnerVolumeSpecName "kube-api-access-djpfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:22.906558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.906529 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.906558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.906554 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72wfj\" (UniqueName: \"kubernetes.io/projected/e4d5532c-91de-471d-bdd7-e9ad1b0ce399-kube-api-access-72wfj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.906558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.906565 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ac031fb-b03c-4a94-9623-e19a1f4296e5-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.906774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.906574 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djpfd\" (UniqueName: \"kubernetes.io/projected/8c1bb6bd-ca03-45c2-b563-0f884658acd0-kube-api-access-djpfd\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.906774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.906582 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1bb6bd-ca03-45c2-b563-0f884658acd0-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.906774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.906592 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cclrt\" (UniqueName: \"kubernetes.io/projected/1ac031fb-b03c-4a94-9623-e19a1f4296e5-kube-api-access-cclrt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:22.913611 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:22.913587 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c01c2c_588b_4480_9e22_c3a71a58ea3c.slice/crio-cde60928bdfc3b360056c11a28f57d928072280c9836a21af7bc7cf42fcca6ea WatchSource:0}: Error finding container cde60928bdfc3b360056c11a28f57d928072280c9836a21af7bc7cf42fcca6ea: Status 404 returned error can't find the container with id cde60928bdfc3b360056c11a28f57d928072280c9836a21af7bc7cf42fcca6ea Oct 30 18:45:22.914437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:22.914412 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst"] Oct 30 18:45:23.064294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.064248 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06cf935f-86dc-4790-93b2-c7baea9b6a42" path="/var/lib/kubelet/pods/06cf935f-86dc-4790-93b2-c7baea9b6a42/volumes" Oct 30 18:45:23.064700 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.064685 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" path="/var/lib/kubelet/pods/bb2168db-5894-4b73-ba78-754f1d8863bc/volumes" Oct 30 18:45:23.065056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.065044 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfffb210-225d-4707-9bdd-4608be2dc749" path="/var/lib/kubelet/pods/cfffb210-225d-4707-9bdd-4608be2dc749/volumes" Oct 30 18:45:23.065736 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.065720 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" path="/var/lib/kubelet/pods/dbf488c3-95ac-4149-9b8b-ebb4522cc909/volumes" Oct 30 18:45:23.066083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.066071 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc189cb4-8886-479c-903a-752c5ad19901" path="/var/lib/kubelet/pods/dc189cb4-8886-479c-903a-752c5ad19901/volumes" Oct 30 18:45:23.066450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.066437 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" path="/var/lib/kubelet/pods/edb49158-4950-4fcb-9ddc-aaa5005a019f/volumes" Oct 30 18:45:23.405389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.405356 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" Oct 30 18:45:23.405817 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.405354 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv" event={"ID":"e4d5532c-91de-471d-bdd7-e9ad1b0ce399","Type":"ContainerDied","Data":"6c476791f158f01c19e69f56b050718c10d2a9a1341926d26d71accb735a8420"} Oct 30 18:45:23.405817 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.405496 4862 scope.go:122] "RemoveContainer" containerID="cf29d31c5704568cf0a0b2fad55943e9c837b4dc3aa7a147439e24c34374fc02" Oct 30 18:45:23.406953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.406817 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" event={"ID":"a97dc6f5-10b4-4d60-90bf-423a8da7ab74","Type":"ContainerDied","Data":"15c286bfd3d4ef561ac5e1188805d15fa28e5f844beb702f44fa8d05f5748ad9"} Oct 30 18:45:23.406953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.406831 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24" Oct 30 18:45:23.410011 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.409992 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" Oct 30 18:45:23.410170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.410075 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg" event={"ID":"1ac031fb-b03c-4a94-9623-e19a1f4296e5","Type":"ContainerDied","Data":"5f19654f06a658e6e9c50925eeba07fa8d2c33465f1ae03265a9c5bb474eacaa"} Oct 30 18:45:23.412469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.412320 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" Oct 30 18:45:23.412469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.412405 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m" event={"ID":"8c1bb6bd-ca03-45c2-b563-0f884658acd0","Type":"ContainerDied","Data":"2e8d9182d7b1b39acac3279521cc253901eb9172c2e99dcfcc840c75a7256a24"} Oct 30 18:45:23.414281 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.414243 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" event={"ID":"74c01c2c-588b-4480-9e22-c3a71a58ea3c","Type":"ContainerStarted","Data":"6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db"} Oct 30 18:45:23.414399 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.414315 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:23.414399 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.414340 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" event={"ID":"74c01c2c-588b-4480-9e22-c3a71a58ea3c","Type":"ContainerStarted","Data":"cde60928bdfc3b360056c11a28f57d928072280c9836a21af7bc7cf42fcca6ea"} Oct 30 18:45:23.432342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.432325 4862 scope.go:122] "RemoveContainer" containerID="2d79ddb7d27ef67efc412bfc5556c8487753f5643f27b4beb451aa6a22157f84" Oct 30 18:45:23.442162 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.442145 4862 scope.go:122] "RemoveContainer" containerID="16ddefd5a24aff31f735bacfdc7dbda5b4860e6438840c9869dc1218c9650611" Oct 30 18:45:23.451289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.451245 4862 scope.go:122] "RemoveContainer" containerID="240252f2cb0ad0ce635b9829f1475ece0f3bb891bfe5ac473632ab34700d9085" Oct 30 18:45:23.506575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.506527 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" podStartSLOduration=1.50651468 podStartE2EDuration="1.50651468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:23.505123622 +0000 UTC m=+2822.554168535" watchObservedRunningTime="2026-10-30 18:45:23.50651468 +0000 UTC m=+2822.555559630" Oct 30 18:45:23.577423 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.577390 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg"] Oct 30 18:45:23.581571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.581550 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-1a2436-29889765-gnsbg"] Oct 30 18:45:23.644704 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.644674 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24"] Oct 30 18:45:23.649200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.649170 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ab244c-29889765-kvj24"] Oct 30 18:45:23.714169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.714092 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv"] Oct 30 18:45:23.717129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.717098 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889765-pcljv"] Oct 30 18:45:23.756862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.756828 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m"] Oct 30 18:45:23.761601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:23.761573 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889765-9z92m"] Oct 30 18:45:24.306612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.306582 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk"] Oct 30 18:45:24.307157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307144 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerName="cleanup" Oct 30 18:45:24.307214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307159 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerName="cleanup" Oct 30 18:45:24.307214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307171 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerName="cleanup" Oct 30 18:45:24.307214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307176 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerName="cleanup" Oct 30 18:45:24.307214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307185 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerName="cleanup" Oct 30 18:45:24.307214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307189 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerName="cleanup" Oct 30 18:45:24.307214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307200 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerName="cleanup" Oct 30 18:45:24.307214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307204 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerName="cleanup" Oct 30 18:45:24.307214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307211 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerName="cleanup" Oct 30 18:45:24.307214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307215 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307223 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307227 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307237 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307241 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307249 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307253 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307281 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307289 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307300 4862 cpu_manager.go:414] "containerMap: removing container" podUID="dc189cb4-8886-479c-903a-752c5ad19901" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307305 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc189cb4-8886-479c-903a-752c5ad19901" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307313 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307321 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2168db-5894-4b73-ba78-754f1d8863bc" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307327 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307331 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307338 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307342 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307351 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307355 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307367 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307371 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307473 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307481 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf488c3-95ac-4149-9b8b-ebb4522cc909" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307491 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307498 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307505 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307512 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc189cb4-8886-479c-903a-752c5ad19901" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307518 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307523 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307529 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c951a41-65ad-48dd-bb7e-1cfde1ef853f" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307535 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307541 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50bc46b-93f1-40f9-a6e3-78bf998aa2ca" containerName="cleanup" Oct 30 18:45:24.307531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307546 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerName="cleanup" Oct 30 18:45:24.308450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307551 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" containerName="cleanup" Oct 30 18:45:24.308450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307556 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerName="cleanup" Oct 30 18:45:24.308450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307563 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" containerName="cleanup" Oct 30 18:45:24.308450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307569 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb49158-4950-4fcb-9ddc-aaa5005a019f" containerName="cleanup" Oct 30 18:45:24.308450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.307577 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2fed97-f53d-4f11-a967-8b975837a13e" containerName="cleanup" Oct 30 18:45:24.312965 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.312946 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.329427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.329410 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"payload-processing-external-model-e2e--541faec9-dockercfg-klj8k\"" Oct 30 18:45:24.343669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.343649 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk"] Oct 30 18:45:24.345463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.345447 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"openshift-service-ca.crt\"" Oct 30 18:45:24.345615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.345596 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"routing-overlay\"" Oct 30 18:45:24.345737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.345725 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"kube-root-ca.crt\"" Oct 30 18:45:24.349647 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.349622 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.349782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.349762 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.349841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.349823 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7df\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-kube-api-access-gv7df\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.349882 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.349869 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.351458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.351439 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d1a330\"" Oct 30 18:45:24.413327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.411991 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc"] Oct 30 18:45:24.413327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.413092 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerName="cleanup" Oct 30 18:45:24.413327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.413115 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerName="cleanup" Oct 30 18:45:24.413327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.413228 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerName="cleanup" Oct 30 18:45:24.413327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.413239 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" containerName="cleanup" Oct 30 18:45:24.433730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.421596 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" containerName="cleanup" Oct 30 18:45:24.448287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.448244 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc"] Oct 30 18:45:24.449253 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.448550 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.451157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.451132 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cdb2c67-tdprc\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.452571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.452553 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxf6x\" (UniqueName: \"kubernetes.io/projected/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-kube-api-access-wxf6x\") pod \"maas-api-revoke-sub-1cdb2c67-tdprc\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.452746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.452728 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.452975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.452948 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.453064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.452994 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7df\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-kube-api-access-gv7df\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.453064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.453036 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.453193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.453172 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cdb2c67-tdprc\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.453340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.453323 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"agc-cred-20961bd79269\"" Oct 30 18:45:24.454378 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.454356 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.458721 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.455891 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.462060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.462037 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.503153 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.503111 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7df\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-kube-api-access-gv7df\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.529803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.529767 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:24.529803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.529789 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:45:24.553959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.553936 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cdb2c67-tdprc\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.554083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.554030 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cdb2c67-tdprc\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.554083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.554057 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxf6x\" (UniqueName: \"kubernetes.io/projected/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-kube-api-access-wxf6x\") pod \"maas-api-revoke-sub-1cdb2c67-tdprc\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.554862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.554825 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cdb2c67-tdprc\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.557147 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.557094 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cdb2c67-tdprc\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.574326 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.574302 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxf6x\" (UniqueName: \"kubernetes.io/projected/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-kube-api-access-wxf6x\") pod \"maas-api-revoke-sub-1cdb2c67-tdprc\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.626977 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.626951 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:24.798753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.798717 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:24.932426 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:24.932402 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk"] Oct 30 18:45:24.932904 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:24.932878 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf85eb34_a08a_4fb5_914f_f8a55df85e04.slice/crio-ef8f46d505236a6c484172d16eb5c36cb2462abbbd9e1d9a75a456e23c199504 WatchSource:0}: Error finding container ef8f46d505236a6c484172d16eb5c36cb2462abbbd9e1d9a75a456e23c199504: Status 404 returned error can't find the container with id ef8f46d505236a6c484172d16eb5c36cb2462abbbd9e1d9a75a456e23c199504 Oct 30 18:45:25.066128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.066033 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac031fb-b03c-4a94-9623-e19a1f4296e5" path="/var/lib/kubelet/pods/1ac031fb-b03c-4a94-9623-e19a1f4296e5/volumes" Oct 30 18:45:25.066811 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.066788 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1bb6bd-ca03-45c2-b563-0f884658acd0" path="/var/lib/kubelet/pods/8c1bb6bd-ca03-45c2-b563-0f884658acd0/volumes" Oct 30 18:45:25.067458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.067438 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97dc6f5-10b4-4d60-90bf-423a8da7ab74" path="/var/lib/kubelet/pods/a97dc6f5-10b4-4d60-90bf-423a8da7ab74/volumes" Oct 30 18:45:25.068468 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.068446 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d5532c-91de-471d-bdd7-e9ad1b0ce399" path="/var/lib/kubelet/pods/e4d5532c-91de-471d-bdd7-e9ad1b0ce399/volumes" Oct 30 18:45:25.107991 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.107970 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc"] Oct 30 18:45:25.108382 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:25.108362 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bb4ab2_9038_4a6d_a3e9_cf2a5c25ff05.slice/crio-6633aa2bc09b465dfa45a6bf609253c6ca93e8e6122f735522b818e714e6e96b WatchSource:0}: Error finding container 6633aa2bc09b465dfa45a6bf609253c6ca93e8e6122f735522b818e714e6e96b: Status 404 returned error can't find the container with id 6633aa2bc09b465dfa45a6bf609253c6ca93e8e6122f735522b818e714e6e96b Oct 30 18:45:25.252045 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:25.252015 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:45:25.252179 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:25.252014 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:45:25.478888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.478860 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" event={"ID":"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05","Type":"ContainerStarted","Data":"00a6f2b7fa737bba611ee660eff4849643a060cfaf50b1e8e593ff06c1d9999e"} Oct 30 18:45:25.479254 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.478924 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" event={"ID":"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05","Type":"ContainerStarted","Data":"6633aa2bc09b465dfa45a6bf609253c6ca93e8e6122f735522b818e714e6e96b"} Oct 30 18:45:25.480595 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.480561 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" event={"ID":"af85eb34-a08a-4fb5-914f-f8a55df85e04","Type":"ContainerStarted","Data":"5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878"} Oct 30 18:45:25.480697 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.480615 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" event={"ID":"af85eb34-a08a-4fb5-914f-f8a55df85e04","Type":"ContainerStarted","Data":"ef8f46d505236a6c484172d16eb5c36cb2462abbbd9e1d9a75a456e23c199504"} Oct 30 18:45:25.480697 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.480644 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:25.529050 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.529011 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" podStartSLOduration=1.529000299 podStartE2EDuration="1.529000299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:25.526338929 +0000 UTC m=+2824.575383842" watchObservedRunningTime="2026-10-30 18:45:25.529000299 +0000 UTC m=+2824.578045211" Oct 30 18:45:25.568703 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:25.568665 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" podStartSLOduration=1.568653879 podStartE2EDuration="1.568653879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:25.566021849 +0000 UTC m=+2824.615066761" watchObservedRunningTime="2026-10-30 18:45:25.568653879 +0000 UTC m=+2824.617698792" Oct 30 18:45:26.489114 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:26.489075 4862 generic.go:356] "Generic (PLEG): container finished" podID="a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05" containerID="00a6f2b7fa737bba611ee660eff4849643a060cfaf50b1e8e593ff06c1d9999e" exitCode=0 Oct 30 18:45:26.489514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:26.489159 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" event={"ID":"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05","Type":"ContainerDied","Data":"00a6f2b7fa737bba611ee660eff4849643a060cfaf50b1e8e593ff06c1d9999e"} Oct 30 18:45:27.666075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.666054 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:27.689949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.689925 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-kube-api-access-wxf6x\" (UniqueName: \"kubernetes.io/projected/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-kube-api-access-wxf6x\") pod \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " Oct 30 18:45:27.690066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.690056 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-openshift-ca\") pod \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " Oct 30 18:45:27.690128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.690084 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-xks-ca\") pod \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\" (UID: \"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05\") " Oct 30 18:45:27.690400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.690374 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-openshift-ca" pod "a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05" (UID: "a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:27.690542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.690526 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:27.692221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.692193 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-xks-ca" pod "a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05" (UID: "a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:27.692221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.692206 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-kube-api-access-wxf6x" pod "a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05" (UID: "a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05"). InnerVolumeSpecName "kube-api-access-wxf6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:27.792130 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.792052 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxf6x\" (UniqueName: \"kubernetes.io/projected/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-kube-api-access-wxf6x\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:27.792130 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:27.792082 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:28.504855 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:28.504825 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" Oct 30 18:45:28.505026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:28.504848 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc" event={"ID":"a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05","Type":"ContainerDied","Data":"6633aa2bc09b465dfa45a6bf609253c6ca93e8e6122f735522b818e714e6e96b"} Oct 30 18:45:28.505026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:28.504921 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6633aa2bc09b465dfa45a6bf609253c6ca93e8e6122f735522b818e714e6e96b" Oct 30 18:45:29.763761 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:29.763729 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:45:31.193080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.192997 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn"] Oct 30 18:45:31.193666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.193649 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05" containerName="revoke-keys" Oct 30 18:45:31.193731 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.193668 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05" containerName="revoke-keys" Oct 30 18:45:31.193795 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.193785 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05" containerName="revoke-keys" Oct 30 18:45:31.197681 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.197664 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.218782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.218750 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn"] Oct 30 18:45:31.225717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.225693 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:31.225863 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.225788 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:45:31.233619 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.233592 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thd65\" (UniqueName: \"kubernetes.io/projected/364a4ab5-d003-44dc-b85c-6d308fe9ed03-kube-api-access-thd65\") pod \"maas-api-revoke-sub-d128ee44-xfwtn\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.233769 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.233750 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d128ee44-xfwtn\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.233855 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.233824 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d128ee44-xfwtn\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.334787 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.334753 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d128ee44-xfwtn\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.335024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.334816 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d128ee44-xfwtn\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.335024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.334964 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thd65\" (UniqueName: \"kubernetes.io/projected/364a4ab5-d003-44dc-b85c-6d308fe9ed03-kube-api-access-thd65\") pod \"maas-api-revoke-sub-d128ee44-xfwtn\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.335679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.335653 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d128ee44-xfwtn\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.337420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.337398 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d128ee44-xfwtn\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.355626 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.355333 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thd65\" (UniqueName: \"kubernetes.io/projected/364a4ab5-d003-44dc-b85c-6d308fe9ed03-kube-api-access-thd65\") pod \"maas-api-revoke-sub-d128ee44-xfwtn\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.517699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.516410 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:31.880400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:31.880367 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn"] Oct 30 18:45:31.880554 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:31.880532 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364a4ab5_d003_44dc_b85c_6d308fe9ed03.slice/crio-b19b524f1bd9d255bcb0b5a60a8fc4db2f794350f22ec9fc831ee3840d53f6fc WatchSource:0}: Error finding container b19b524f1bd9d255bcb0b5a60a8fc4db2f794350f22ec9fc831ee3840d53f6fc: Status 404 returned error can't find the container with id b19b524f1bd9d255bcb0b5a60a8fc4db2f794350f22ec9fc831ee3840d53f6fc Oct 30 18:45:32.494560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.494491 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg"] Oct 30 18:45:32.498817 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.498794 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.513922 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.513897 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"agc-cred-157fc3a04bb6\"" Oct 30 18:45:32.517479 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.517458 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg"] Oct 30 18:45:32.549294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.549250 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztxp6\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-kube-api-access-ztxp6\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.549430 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.549349 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.549474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.549429 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.549571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.549548 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.550793 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.550769 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" event={"ID":"364a4ab5-d003-44dc-b85c-6d308fe9ed03","Type":"ContainerStarted","Data":"99f04fbbe2cf0d18a2dbbbd36ea94fb5a2deab7bd92f579e63841bc7be3e02fd"} Oct 30 18:45:32.550895 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.550814 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" event={"ID":"364a4ab5-d003-44dc-b85c-6d308fe9ed03","Type":"ContainerStarted","Data":"b19b524f1bd9d255bcb0b5a60a8fc4db2f794350f22ec9fc831ee3840d53f6fc"} Oct 30 18:45:32.592981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.592942 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" podStartSLOduration=1.592928935 podStartE2EDuration="1.592928935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:32.591467392 +0000 UTC m=+2831.640512305" watchObservedRunningTime="2026-10-30 18:45:32.592928935 +0000 UTC m=+2831.641973850" Oct 30 18:45:32.650514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.650491 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxp6\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-kube-api-access-ztxp6\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.650646 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.650525 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.650646 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.650588 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.650646 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.650623 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.651407 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.651383 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.651481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.651467 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.653195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.653176 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.673406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.673382 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxp6\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-kube-api-access-ztxp6\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:32.814482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:32.814459 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:33.118960 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:33.118934 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c622f1_e75a_4ca4_a22b_89acd70f0822.slice/crio-e7a054f0e033683fbe6c1cb5b45e265646133bd4e4ca05dba2195cff9b19075f WatchSource:0}: Error finding container e7a054f0e033683fbe6c1cb5b45e265646133bd4e4ca05dba2195cff9b19075f: Status 404 returned error can't find the container with id e7a054f0e033683fbe6c1cb5b45e265646133bd4e4ca05dba2195cff9b19075f Oct 30 18:45:33.119188 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:33.119168 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg"] Oct 30 18:45:33.560931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:33.560899 4862 generic.go:356] "Generic (PLEG): container finished" podID="364a4ab5-d003-44dc-b85c-6d308fe9ed03" containerID="99f04fbbe2cf0d18a2dbbbd36ea94fb5a2deab7bd92f579e63841bc7be3e02fd" exitCode=0 Oct 30 18:45:33.561431 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:33.560995 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" event={"ID":"364a4ab5-d003-44dc-b85c-6d308fe9ed03","Type":"ContainerDied","Data":"99f04fbbe2cf0d18a2dbbbd36ea94fb5a2deab7bd92f579e63841bc7be3e02fd"} Oct 30 18:45:33.563190 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:33.563159 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" event={"ID":"e3c622f1-e75a-4ca4-a22b-89acd70f0822","Type":"ContainerStarted","Data":"7acac5ccdce081648abba1de15a7f74b959adc871c13d3752cad2307478a1f21"} Oct 30 18:45:33.563354 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:33.563213 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" event={"ID":"e3c622f1-e75a-4ca4-a22b-89acd70f0822","Type":"ContainerStarted","Data":"e7a054f0e033683fbe6c1cb5b45e265646133bd4e4ca05dba2195cff9b19075f"} Oct 30 18:45:33.563354 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:33.563242 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:33.649453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:33.649399 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" podStartSLOduration=1.64938534 podStartE2EDuration="1.64938534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:33.646065629 +0000 UTC m=+2832.695110542" watchObservedRunningTime="2026-10-30 18:45:33.64938534 +0000 UTC m=+2832.698430290" Oct 30 18:45:34.474094 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.474067 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:34.551013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.550978 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf"] Oct 30 18:45:34.739219 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.739169 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:34.785916 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.785887 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364a4ab5-d003-44dc-b85c-6d308fe9ed03-kube-api-access-thd65\" (UniqueName: \"kubernetes.io/projected/364a4ab5-d003-44dc-b85c-6d308fe9ed03-kube-api-access-thd65\") pod \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " Oct 30 18:45:34.786041 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.785970 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-openshift-ca\") pod \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " Oct 30 18:45:34.786041 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.785995 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-xks-ca\") pod \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\" (UID: \"364a4ab5-d003-44dc-b85c-6d308fe9ed03\") " Oct 30 18:45:34.786404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.786377 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-openshift-ca" pod "364a4ab5-d003-44dc-b85c-6d308fe9ed03" (UID: "364a4ab5-d003-44dc-b85c-6d308fe9ed03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:34.786572 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.786551 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:34.788063 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.788037 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364a4ab5-d003-44dc-b85c-6d308fe9ed03-kube-api-access-thd65" pod "364a4ab5-d003-44dc-b85c-6d308fe9ed03" (UID: "364a4ab5-d003-44dc-b85c-6d308fe9ed03"). InnerVolumeSpecName "kube-api-access-thd65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:34.788173 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.788135 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-xks-ca" pod "364a4ab5-d003-44dc-b85c-6d308fe9ed03" (UID: "364a4ab5-d003-44dc-b85c-6d308fe9ed03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:34.887803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.887777 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/364a4ab5-d003-44dc-b85c-6d308fe9ed03-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:34.887803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:34.887802 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thd65\" (UniqueName: \"kubernetes.io/projected/364a4ab5-d003-44dc-b85c-6d308fe9ed03-kube-api-access-thd65\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:35.582531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:35.582498 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" event={"ID":"364a4ab5-d003-44dc-b85c-6d308fe9ed03","Type":"ContainerDied","Data":"b19b524f1bd9d255bcb0b5a60a8fc4db2f794350f22ec9fc831ee3840d53f6fc"} Oct 30 18:45:35.582698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:35.582552 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b19b524f1bd9d255bcb0b5a60a8fc4db2f794350f22ec9fc831ee3840d53f6fc" Oct 30 18:45:35.582698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:35.582509 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn" Oct 30 18:45:35.811988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:35.811950 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5"] Oct 30 18:45:36.493600 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:36.493572 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:37.678084 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:37.678044 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:45:38.385754 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.385721 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h"] Oct 30 18:45:38.386415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.386399 4862 cpu_manager.go:414] "containerMap: removing container" podUID="364a4ab5-d003-44dc-b85c-6d308fe9ed03" containerName="revoke-keys" Oct 30 18:45:38.386489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.386416 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364a4ab5-d003-44dc-b85c-6d308fe9ed03" containerName="revoke-keys" Oct 30 18:45:38.386546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.386531 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="364a4ab5-d003-44dc-b85c-6d308fe9ed03" containerName="revoke-keys" Oct 30 18:45:38.391216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.391200 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.424360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.424317 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h"] Oct 30 18:45:38.543125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.543097 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.543317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.543294 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g58sw\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-kube-api-access-g58sw\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.543468 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.543448 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.543539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.543494 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.646598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.646472 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.647599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.647569 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.647799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.647483 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.648146 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.648121 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.648302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.647615 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.648408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.648389 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g58sw\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-kube-api-access-g58sw\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.672285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.671305 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g58sw\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-kube-api-access-g58sw\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.691288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.688923 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-5997z884h\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.709288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.706339 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:38.849552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:38.849507 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk"] Oct 30 18:45:39.046245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.046216 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h"] Oct 30 18:45:39.046991 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:39.046969 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d19e264_9bc0_4640_acda_d9825ee65692.slice/crio-7a699fb6d62c447805daec07e3171a42df8fc703664676c4c6471b195acce670 WatchSource:0}: Error finding container 7a699fb6d62c447805daec07e3171a42df8fc703664676c4c6471b195acce670: Status 404 returned error can't find the container with id 7a699fb6d62c447805daec07e3171a42df8fc703664676c4c6471b195acce670 Oct 30 18:45:39.551768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.551731 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" podUID="f9870774-1319-40f7-92d9-74138973540f" containerName="payload-processing" containerID="cri-o://43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd" gracePeriod=25 Oct 30 18:45:39.611988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.611957 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" event={"ID":"6d19e264-9bc0-4640-acda-d9825ee65692","Type":"ContainerStarted","Data":"5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12"} Oct 30 18:45:39.612124 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.612012 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" event={"ID":"6d19e264-9bc0-4640-acda-d9825ee65692","Type":"ContainerStarted","Data":"7a699fb6d62c447805daec07e3171a42df8fc703664676c4c6471b195acce670"} Oct 30 18:45:39.612124 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.612052 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:39.655370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.655325 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" podStartSLOduration=1.655307807 podStartE2EDuration="1.655307807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:39.652484771 +0000 UTC m=+2838.701529684" watchObservedRunningTime="2026-10-30 18:45:39.655307807 +0000 UTC m=+2838.704352721" Oct 30 18:45:39.780515 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:39.780480 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache]" Oct 30 18:45:39.822018 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.821962 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:45:39.982735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.982702 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-routing-overlay-volume\") pod \"f9870774-1319-40f7-92d9-74138973540f\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " Oct 30 18:45:39.982983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.982965 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-kube-api-access-bxjnv\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-kube-api-access-bxjnv\") pod \"f9870774-1319-40f7-92d9-74138973540f\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " Oct 30 18:45:39.983878 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.983850 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-plugins-config-volume\") pod \"f9870774-1319-40f7-92d9-74138973540f\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " Oct 30 18:45:39.984211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.984189 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-provider-credentials\") pod \"f9870774-1319-40f7-92d9-74138973540f\" (UID: \"f9870774-1319-40f7-92d9-74138973540f\") " Oct 30 18:45:39.986753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.986497 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-routing-overlay-volume" pod "f9870774-1319-40f7-92d9-74138973540f" (UID: "f9870774-1319-40f7-92d9-74138973540f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:39.986974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.986949 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-plugins-config-volume" pod "f9870774-1319-40f7-92d9-74138973540f" (UID: "f9870774-1319-40f7-92d9-74138973540f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:39.987760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.987738 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:39.988061 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.988038 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-kube-api-access-bxjnv" pod "f9870774-1319-40f7-92d9-74138973540f" (UID: "f9870774-1319-40f7-92d9-74138973540f"). InnerVolumeSpecName "kube-api-access-bxjnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:39.989859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:39.989837 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-provider-credentials" pod "f9870774-1319-40f7-92d9-74138973540f" (UID: "f9870774-1319-40f7-92d9-74138973540f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:40.089113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.089045 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:40.089113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.089069 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxjnv\" (UniqueName: \"kubernetes.io/projected/f9870774-1319-40f7-92d9-74138973540f-kube-api-access-bxjnv\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:40.089113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.089079 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9870774-1319-40f7-92d9-74138973540f-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:40.620760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.620723 4862 generic.go:356] "Generic (PLEG): container finished" podID="f9870774-1319-40f7-92d9-74138973540f" containerID="43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd" exitCode=0 Oct 30 18:45:40.620925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.620775 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" Oct 30 18:45:40.620925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.620812 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" event={"ID":"f9870774-1319-40f7-92d9-74138973540f","Type":"ContainerDied","Data":"43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd"} Oct 30 18:45:40.620925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.620871 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf" event={"ID":"f9870774-1319-40f7-92d9-74138973540f","Type":"ContainerDied","Data":"68a987ebcbba798c1bb0cea6df5504087dfc64ee4cbb552ceff92b9b0d429278"} Oct 30 18:45:40.620925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.620894 4862 scope.go:122] "RemoveContainer" containerID="43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd" Oct 30 18:45:40.631830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.631814 4862 scope.go:122] "RemoveContainer" containerID="43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd" Oct 30 18:45:40.632077 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:40.632061 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd\": container with ID starting with 43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd not found: ID does not exist" containerID="43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd" Oct 30 18:45:40.632123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.632084 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd"} err="failed to get container status \"43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd\": rpc error: code = NotFound desc = could not find container \"43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd\": container with ID starting with 43748052d8101e5c3ae7aa3839e8e210d4d8ce9e793edb7e82da941e644b74fd not found: ID does not exist" Oct 30 18:45:40.673007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.672986 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf"] Oct 30 18:45:40.685062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.685039 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997m92zf"] Oct 30 18:45:40.812886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:40.812854 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" podUID="2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" containerName="payload-processing" containerID="cri-o://b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005" gracePeriod=25 Oct 30 18:45:41.063851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.063817 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9870774-1319-40f7-92d9-74138973540f" path="/var/lib/kubelet/pods/f9870774-1319-40f7-92d9-74138973540f/volumes" Oct 30 18:45:41.097687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.097666 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:45:41.202688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.202618 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-kube-api-access-czl4z\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-kube-api-access-czl4z\") pod \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " Oct 30 18:45:41.202688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.202651 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-routing-overlay-volume\") pod \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " Oct 30 18:45:41.202688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.202688 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-plugins-config-volume\") pod \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " Oct 30 18:45:41.202976 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.202717 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-provider-credentials\") pod \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\" (UID: \"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28\") " Oct 30 18:45:41.203107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.203074 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-routing-overlay-volume" pod "2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" (UID: "2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:41.203107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.203091 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-plugins-config-volume" pod "2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" (UID: "2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:41.203400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.203377 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:41.203502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.203473 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:41.205001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.204980 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-kube-api-access-czl4z" pod "2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" (UID: "2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28"). InnerVolumeSpecName "kube-api-access-czl4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:41.205102 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.205027 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-provider-credentials" pod "2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" (UID: "2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:41.304434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.304409 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czl4z\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-kube-api-access-czl4z\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:41.304434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.304433 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:41.630388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.630359 4862 generic.go:356] "Generic (PLEG): container finished" podID="2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" containerID="b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005" exitCode=0 Oct 30 18:45:41.630564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.630405 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" Oct 30 18:45:41.630564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.630438 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" event={"ID":"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28","Type":"ContainerDied","Data":"b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005"} Oct 30 18:45:41.630564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.630501 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5" event={"ID":"2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28","Type":"ContainerDied","Data":"823174dcb8a90ff765f297534f79636d1de4dcc86a11ff4edf1a2b54dcf0c54a"} Oct 30 18:45:41.630564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.630536 4862 scope.go:122] "RemoveContainer" containerID="b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005" Oct 30 18:45:41.641291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.641250 4862 scope.go:122] "RemoveContainer" containerID="b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005" Oct 30 18:45:41.641641 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:41.641622 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005\": container with ID starting with b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005 not found: ID does not exist" containerID="b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005" Oct 30 18:45:41.641705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.641649 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005"} err="failed to get container status \"b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005\": rpc error: code = NotFound desc = could not find container \"b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005\": container with ID starting with b57ca6bcb5ee5a2a009ca84c77b0bf32b3c0f9700a9982a4510cd55bdd5d5005 not found: ID does not exist" Oct 30 18:45:41.683898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.683872 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5"] Oct 30 18:45:41.695973 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:41.695953 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-8477lqjr5"] Oct 30 18:45:43.062968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.062934 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" path="/var/lib/kubelet/pods/2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28/volumes" Oct 30 18:45:43.520438 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.520363 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg"] Oct 30 18:45:43.522969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.522943 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:43.560900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.560876 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg"] Oct 30 18:45:43.561486 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.561470 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" containerName="payload-processing" Oct 30 18:45:43.561558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.561487 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" containerName="payload-processing" Oct 30 18:45:43.561558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.561525 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f9870774-1319-40f7-92d9-74138973540f" containerName="payload-processing" Oct 30 18:45:43.561558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.561531 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9870774-1319-40f7-92d9-74138973540f" containerName="payload-processing" Oct 30 18:45:43.561718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.561625 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9870774-1319-40f7-92d9-74138973540f" containerName="payload-processing" Oct 30 18:45:43.561718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.561638 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf5b6b7-eb3c-49ae-9cb5-ae7ec8b76f28" containerName="payload-processing" Oct 30 18:45:43.566985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.566965 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.585605 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.585586 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-713dc8\"/\"agc-cred-31f2776bb2c6\"" Oct 30 18:45:43.588900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.588881 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg"] Oct 30 18:45:43.636759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.636720 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.636940 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.636836 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.638039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.638005 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw72p\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-kube-api-access-tw72p\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.638234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.638217 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.740182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.740130 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.740441 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.740315 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.740441 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.740378 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.740441 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.740425 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw72p\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-kube-api-access-tw72p\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.740953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.740922 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.741164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.741136 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.743804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.743774 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.765903 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.765851 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw72p\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-kube-api-access-tw72p\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:43.850720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.850682 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" podUID="af85eb34-a08a-4fb5-914f-f8a55df85e04" containerName="payload-processing" containerID="cri-o://5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878" gracePeriod=25 Oct 30 18:45:43.882423 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:43.882390 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:44.203314 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.203279 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:44.269919 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.269893 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg"] Oct 30 18:45:44.270070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.270049 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-routing-overlay-volume\") pod \"af85eb34-a08a-4fb5-914f-f8a55df85e04\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " Oct 30 18:45:44.270180 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.270165 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-kube-api-access-gv7df\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-kube-api-access-gv7df\") pod \"af85eb34-a08a-4fb5-914f-f8a55df85e04\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " Oct 30 18:45:44.270250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.270230 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-provider-credentials\") pod \"af85eb34-a08a-4fb5-914f-f8a55df85e04\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " Oct 30 18:45:44.270399 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.270346 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-plugins-config-volume\") pod \"af85eb34-a08a-4fb5-914f-f8a55df85e04\" (UID: \"af85eb34-a08a-4fb5-914f-f8a55df85e04\") " Oct 30 18:45:44.270858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.270830 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-routing-overlay-volume" pod "af85eb34-a08a-4fb5-914f-f8a55df85e04" (UID: "af85eb34-a08a-4fb5-914f-f8a55df85e04"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:44.271201 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.270972 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:44.271201 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.271162 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-plugins-config-volume" pod "af85eb34-a08a-4fb5-914f-f8a55df85e04" (UID: "af85eb34-a08a-4fb5-914f-f8a55df85e04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:44.273116 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:44.273074 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00b9600_af60_49ca_b66f_d8f70e66afdc.slice/crio-ba8f9ebd754ddce81d703b19c264c687afdc97b8c4cce0342805a5f3b2d0a93c WatchSource:0}: Error finding container ba8f9ebd754ddce81d703b19c264c687afdc97b8c4cce0342805a5f3b2d0a93c: Status 404 returned error can't find the container with id ba8f9ebd754ddce81d703b19c264c687afdc97b8c4cce0342805a5f3b2d0a93c Oct 30 18:45:44.273233 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.273130 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-provider-credentials" pod "af85eb34-a08a-4fb5-914f-f8a55df85e04" (UID: "af85eb34-a08a-4fb5-914f-f8a55df85e04"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:44.273233 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.273196 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-kube-api-access-gv7df" pod "af85eb34-a08a-4fb5-914f-f8a55df85e04" (UID: "af85eb34-a08a-4fb5-914f-f8a55df85e04"). InnerVolumeSpecName "kube-api-access-gv7df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:44.372673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.372644 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85eb34-a08a-4fb5-914f-f8a55df85e04-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:44.372673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.372670 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7df\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-kube-api-access-gv7df\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:44.372673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.372680 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af85eb34-a08a-4fb5-914f-f8a55df85e04-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:44.432934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.432840 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw"] Oct 30 18:45:44.466971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.433867 4862 cpu_manager.go:414] "containerMap: removing container" podUID="af85eb34-a08a-4fb5-914f-f8a55df85e04" containerName="payload-processing" Oct 30 18:45:44.466971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.433892 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af85eb34-a08a-4fb5-914f-f8a55df85e04" containerName="payload-processing" Oct 30 18:45:44.466971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.434108 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="af85eb34-a08a-4fb5-914f-f8a55df85e04" containerName="payload-processing" Oct 30 18:45:44.490284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.480458 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd"] Oct 30 18:45:44.490284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.484455 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.517277 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.508585 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:45:44.517277 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.509659 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:44.533972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.531791 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw"] Oct 30 18:45:44.533972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.531840 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd"] Oct 30 18:45:44.533972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.532021 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.591591 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.591558 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cae8e5e-dpksw\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.591743 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.591597 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngc56\" (UniqueName: \"kubernetes.io/projected/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-kube-api-access-ngc56\") pod \"maas-api-revoke-sub-6cae8e5e-dpksw\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.591743 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.591684 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.591823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.591747 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.591823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.591800 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.591936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.591899 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66cn\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-kube-api-access-h66cn\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.591936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.591925 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cae8e5e-dpksw\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.667713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.667679 4862 generic.go:356] "Generic (PLEG): container finished" podID="af85eb34-a08a-4fb5-914f-f8a55df85e04" containerID="5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878" exitCode=0 Oct 30 18:45:44.667907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.667731 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" Oct 30 18:45:44.667907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.667767 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" event={"ID":"af85eb34-a08a-4fb5-914f-f8a55df85e04","Type":"ContainerDied","Data":"5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878"} Oct 30 18:45:44.667907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.667828 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk" event={"ID":"af85eb34-a08a-4fb5-914f-f8a55df85e04","Type":"ContainerDied","Data":"ef8f46d505236a6c484172d16eb5c36cb2462abbbd9e1d9a75a456e23c199504"} Oct 30 18:45:44.667907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.667859 4862 scope.go:122] "RemoveContainer" containerID="5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878" Oct 30 18:45:44.669812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.669788 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" event={"ID":"c00b9600-af60-49ca-b66f-d8f70e66afdc","Type":"ContainerStarted","Data":"439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598"} Oct 30 18:45:44.669925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.669830 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" event={"ID":"c00b9600-af60-49ca-b66f-d8f70e66afdc","Type":"ContainerStarted","Data":"ba8f9ebd754ddce81d703b19c264c687afdc97b8c4cce0342805a5f3b2d0a93c"} Oct 30 18:45:44.669981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.669933 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:44.679120 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.679099 4862 scope.go:122] "RemoveContainer" containerID="5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878" Oct 30 18:45:44.679509 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:44.679472 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878\": container with ID starting with 5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878 not found: ID does not exist" containerID="5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878" Oct 30 18:45:44.679604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.679521 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878"} err="failed to get container status \"5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878\": rpc error: code = NotFound desc = could not find container \"5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878\": container with ID starting with 5d84b91027d143cd985f8755e81ba1da2151bc20202beadb4459cda5db119878 not found: ID does not exist" Oct 30 18:45:44.692913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.692864 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66cn\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-kube-api-access-h66cn\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.692913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.692899 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cae8e5e-dpksw\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.693034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.692991 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cae8e5e-dpksw\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.693034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.693016 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngc56\" (UniqueName: \"kubernetes.io/projected/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-kube-api-access-ngc56\") pod \"maas-api-revoke-sub-6cae8e5e-dpksw\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.693126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.693093 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.693126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.693121 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.693209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.693167 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.693574 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.693548 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cae8e5e-dpksw\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.693799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.693781 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.693866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.693836 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.695569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.695554 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cae8e5e-dpksw\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.695910 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.695893 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.712609 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.712587 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngc56\" (UniqueName: \"kubernetes.io/projected/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-kube-api-access-ngc56\") pod \"maas-api-revoke-sub-6cae8e5e-dpksw\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.712696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.712589 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66cn\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-kube-api-access-h66cn\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:44.724217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.724181 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" podStartSLOduration=1.7241688339999999 podStartE2EDuration="1.724168834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:44.722852351 +0000 UTC m=+2843.771897277" watchObservedRunningTime="2026-10-30 18:45:44.724168834 +0000 UTC m=+2843.773213737" Oct 30 18:45:44.760189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.760154 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk"] Oct 30 18:45:44.763413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.763393 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-6bfd7qbtk"] Oct 30 18:45:44.837220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.837187 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:44.850740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:44.850714 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:45.062855 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.062812 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af85eb34-a08a-4fb5-914f-f8a55df85e04" path="/var/lib/kubelet/pods/af85eb34-a08a-4fb5-914f-f8a55df85e04/volumes" Oct 30 18:45:45.144789 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.144763 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw"] Oct 30 18:45:45.146151 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:45.146125 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b0957a_bbbd_44fa_b58c_00ec8d15cd10.slice/crio-0a862dd8b69c803a3a967f6016ae49c288ea4b006509521f9fd9388ceed4808f WatchSource:0}: Error finding container 0a862dd8b69c803a3a967f6016ae49c288ea4b006509521f9fd9388ceed4808f: Status 404 returned error can't find the container with id 0a862dd8b69c803a3a967f6016ae49c288ea4b006509521f9fd9388ceed4808f Oct 30 18:45:45.169610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.169585 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd"] Oct 30 18:45:45.180316 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:45.180287 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ab924d_c201_4f17_9f90_82845d9ca8d5.slice/crio-ce317f4b3c922615a7e26fb5f6d5135d146906ee604c0c316237747ee5130768 WatchSource:0}: Error finding container ce317f4b3c922615a7e26fb5f6d5135d146906ee604c0c316237747ee5130768: Status 404 returned error can't find the container with id ce317f4b3c922615a7e26fb5f6d5135d146906ee604c0c316237747ee5130768 Oct 30 18:45:45.688452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.688409 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" event={"ID":"f5ab924d-c201-4f17-9f90-82845d9ca8d5","Type":"ContainerStarted","Data":"bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739"} Oct 30 18:45:45.688911 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.688814 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:45.688911 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.688855 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" event={"ID":"f5ab924d-c201-4f17-9f90-82845d9ca8d5","Type":"ContainerStarted","Data":"ce317f4b3c922615a7e26fb5f6d5135d146906ee604c0c316237747ee5130768"} Oct 30 18:45:45.690410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.690379 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" event={"ID":"76b0957a-bbbd-44fa-b58c-00ec8d15cd10","Type":"ContainerStarted","Data":"94c5f5fbaa5f8d8fec97acd3be6ce30435296e9a95d07f77e01c663a75bda4f0"} Oct 30 18:45:45.690530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.690432 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" event={"ID":"76b0957a-bbbd-44fa-b58c-00ec8d15cd10","Type":"ContainerStarted","Data":"0a862dd8b69c803a3a967f6016ae49c288ea4b006509521f9fd9388ceed4808f"} Oct 30 18:45:45.729232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.729182 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" podStartSLOduration=1.7291661820000002 podStartE2EDuration="1.729166182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:45.727672405 +0000 UTC m=+2844.776717320" watchObservedRunningTime="2026-10-30 18:45:45.729166182 +0000 UTC m=+2844.778211097" Oct 30 18:45:45.766247 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:45.766207 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" podStartSLOduration=1.766196609 podStartE2EDuration="1.766196609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:45.764784994 +0000 UTC m=+2844.813829907" watchObservedRunningTime="2026-10-30 18:45:45.766196609 +0000 UTC m=+2844.815241522" Oct 30 18:45:46.703000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:46.702968 4862 generic.go:356] "Generic (PLEG): container finished" podID="76b0957a-bbbd-44fa-b58c-00ec8d15cd10" containerID="94c5f5fbaa5f8d8fec97acd3be6ce30435296e9a95d07f77e01c663a75bda4f0" exitCode=0 Oct 30 18:45:46.703397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:46.703050 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" event={"ID":"76b0957a-bbbd-44fa-b58c-00ec8d15cd10","Type":"ContainerDied","Data":"94c5f5fbaa5f8d8fec97acd3be6ce30435296e9a95d07f77e01c663a75bda4f0"} Oct 30 18:45:47.888349 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:47.888252 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:47.929854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:47.929824 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-openshift-ca\") pod \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " Oct 30 18:45:47.930038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:47.929871 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-xks-ca\") pod \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " Oct 30 18:45:47.930038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:47.929946 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-kube-api-access-ngc56\" (UniqueName: \"kubernetes.io/projected/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-kube-api-access-ngc56\") pod \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\" (UID: \"76b0957a-bbbd-44fa-b58c-00ec8d15cd10\") " Oct 30 18:45:47.930215 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:47.930193 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-openshift-ca" pod "76b0957a-bbbd-44fa-b58c-00ec8d15cd10" (UID: "76b0957a-bbbd-44fa-b58c-00ec8d15cd10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:47.930659 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:47.930633 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:47.935709 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:47.935686 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-xks-ca" pod "76b0957a-bbbd-44fa-b58c-00ec8d15cd10" (UID: "76b0957a-bbbd-44fa-b58c-00ec8d15cd10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:47.935826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:47.935752 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-kube-api-access-ngc56" pod "76b0957a-bbbd-44fa-b58c-00ec8d15cd10" (UID: "76b0957a-bbbd-44fa-b58c-00ec8d15cd10"). InnerVolumeSpecName "kube-api-access-ngc56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:48.032228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.032153 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:48.032228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.032182 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngc56\" (UniqueName: \"kubernetes.io/projected/76b0957a-bbbd-44fa-b58c-00ec8d15cd10-kube-api-access-ngc56\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:48.521111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.521064 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" podUID="e3c622f1-e75a-4ca4-a22b-89acd70f0822" containerName="payload-processing" containerID="cri-o://7acac5ccdce081648abba1de15a7f74b959adc871c13d3752cad2307478a1f21" gracePeriod=25 Oct 30 18:45:48.723022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.722987 4862 generic.go:356] "Generic (PLEG): container finished" podID="e3c622f1-e75a-4ca4-a22b-89acd70f0822" containerID="7acac5ccdce081648abba1de15a7f74b959adc871c13d3752cad2307478a1f21" exitCode=0 Oct 30 18:45:48.723173 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.723025 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" event={"ID":"e3c622f1-e75a-4ca4-a22b-89acd70f0822","Type":"ContainerDied","Data":"7acac5ccdce081648abba1de15a7f74b959adc871c13d3752cad2307478a1f21"} Oct 30 18:45:48.724860 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.724840 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" Oct 30 18:45:48.725007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.724870 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw" event={"ID":"76b0957a-bbbd-44fa-b58c-00ec8d15cd10","Type":"ContainerDied","Data":"0a862dd8b69c803a3a967f6016ae49c288ea4b006509521f9fd9388ceed4808f"} Oct 30 18:45:48.725007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.724918 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a862dd8b69c803a3a967f6016ae49c288ea4b006509521f9fd9388ceed4808f" Oct 30 18:45:48.770707 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.770685 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:48.842807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.842780 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-kube-api-access-ztxp6\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-kube-api-access-ztxp6\") pod \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " Oct 30 18:45:48.842989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.842941 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-plugins-config-volume\") pod \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " Oct 30 18:45:48.843123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.843083 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-provider-credentials\") pod \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " Oct 30 18:45:48.843199 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.843182 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-routing-overlay-volume\") pod \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\" (UID: \"e3c622f1-e75a-4ca4-a22b-89acd70f0822\") " Oct 30 18:45:48.843476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.843447 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-plugins-config-volume" pod "e3c622f1-e75a-4ca4-a22b-89acd70f0822" (UID: "e3c622f1-e75a-4ca4-a22b-89acd70f0822"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:48.843826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.843795 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-routing-overlay-volume" pod "e3c622f1-e75a-4ca4-a22b-89acd70f0822" (UID: "e3c622f1-e75a-4ca4-a22b-89acd70f0822"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:48.843957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.843862 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:48.845688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.845662 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-provider-credentials" pod "e3c622f1-e75a-4ca4-a22b-89acd70f0822" (UID: "e3c622f1-e75a-4ca4-a22b-89acd70f0822"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:48.845789 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.845748 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-kube-api-access-ztxp6" pod "e3c622f1-e75a-4ca4-a22b-89acd70f0822" (UID: "e3c622f1-e75a-4ca4-a22b-89acd70f0822"). InnerVolumeSpecName "kube-api-access-ztxp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:48.944827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.944796 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:48.944827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.944827 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3c622f1-e75a-4ca4-a22b-89acd70f0822-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:48.945327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:48.944837 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztxp6\" (UniqueName: \"kubernetes.io/projected/e3c622f1-e75a-4ca4-a22b-89acd70f0822-kube-api-access-ztxp6\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:49.750683 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:49.750649 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" Oct 30 18:45:49.750683 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:49.750654 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg" event={"ID":"e3c622f1-e75a-4ca4-a22b-89acd70f0822","Type":"ContainerDied","Data":"e7a054f0e033683fbe6c1cb5b45e265646133bd4e4ca05dba2195cff9b19075f"} Oct 30 18:45:49.750935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:49.750728 4862 scope.go:122] "RemoveContainer" containerID="7acac5ccdce081648abba1de15a7f74b959adc871c13d3752cad2307478a1f21" Oct 30 18:45:49.808926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:49.808895 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg"] Oct 30 18:45:49.820702 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:49.820678 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbmj4vg"] Oct 30 18:45:49.882740 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:49.882690 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:45:49.886462 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:49.886420 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:45:50.624449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.624423 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:45:50.696399 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.696369 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst"] Oct 30 18:45:50.766985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.766954 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6"] Oct 30 18:45:50.767625 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.767605 4862 cpu_manager.go:414] "containerMap: removing container" podUID="76b0957a-bbbd-44fa-b58c-00ec8d15cd10" containerName="revoke-keys" Oct 30 18:45:50.767625 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.767625 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b0957a-bbbd-44fa-b58c-00ec8d15cd10" containerName="revoke-keys" Oct 30 18:45:50.767794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.767658 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e3c622f1-e75a-4ca4-a22b-89acd70f0822" containerName="payload-processing" Oct 30 18:45:50.767794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.767667 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c622f1-e75a-4ca4-a22b-89acd70f0822" containerName="payload-processing" Oct 30 18:45:50.767901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.767871 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b0957a-bbbd-44fa-b58c-00ec8d15cd10" containerName="revoke-keys" Oct 30 18:45:50.767901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.767890 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c622f1-e75a-4ca4-a22b-89acd70f0822" containerName="payload-processing" Oct 30 18:45:50.772735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.772713 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:50.787448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.787424 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:50.787584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.787477 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:45:50.788680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.788655 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6"] Oct 30 18:45:50.870610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.870575 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8994c\" (UniqueName: \"kubernetes.io/projected/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-kube-api-access-8994c\") pod \"maas-api-revoke-sub-fd7982b6-ppbd6\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:50.870779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.870706 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7982b6-ppbd6\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:50.870846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.870785 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7982b6-ppbd6\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:50.971837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.971769 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8994c\" (UniqueName: \"kubernetes.io/projected/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-kube-api-access-8994c\") pod \"maas-api-revoke-sub-fd7982b6-ppbd6\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:50.971837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.971834 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7982b6-ppbd6\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:50.972034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.971907 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7982b6-ppbd6\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:50.972489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.972471 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7982b6-ppbd6\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:50.974424 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.974403 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7982b6-ppbd6\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:50.991090 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:50.991068 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8994c\" (UniqueName: \"kubernetes.io/projected/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-kube-api-access-8994c\") pod \"maas-api-revoke-sub-fd7982b6-ppbd6\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:51.063409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.063382 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c622f1-e75a-4ca4-a22b-89acd70f0822" path="/var/lib/kubelet/pods/e3c622f1-e75a-4ca4-a22b-89acd70f0822/volumes" Oct 30 18:45:51.087098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.087076 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:51.395029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.395001 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6"] Oct 30 18:45:51.481807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.481784 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp"] Oct 30 18:45:51.487479 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.487461 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.504559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.504418 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-12698c3c4819\"" Oct 30 18:45:51.504559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.504418 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 18:45:51.504559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.504457 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 18:45:51.504559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.504490 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 18:45:51.504559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.504508 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dnczl\"" Oct 30 18:45:51.504559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.504537 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 18:45:51.504953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.504895 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp"] Oct 30 18:45:51.580141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.580110 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8r6c\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-kube-api-access-c8r6c\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.580286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.580239 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-provider-credentials\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.580348 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.580333 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-routing-overlay-volume\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.580412 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.580397 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-plugins-config-volume\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.681460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.681357 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-provider-credentials\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.681879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.681467 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-routing-overlay-volume\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.681879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.681503 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-plugins-config-volume\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.681879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.681589 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8r6c\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-kube-api-access-c8r6c\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.682220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.682153 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-routing-overlay-volume\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.682469 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.682319 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-plugins-config-volume\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.684142 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.684121 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-provider-credentials\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.710817 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.710792 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8r6c\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-kube-api-access-c8r6c\") pod \"payload-processing-external-model-78bf69cc96-bscgp\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.769779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.769752 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" event={"ID":"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3","Type":"ContainerStarted","Data":"8accace7a2770a93927b3de3256dc5fbc751d206e75e8aa65a1ec4aba8a3668c"} Oct 30 18:45:51.769925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.769808 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" event={"ID":"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3","Type":"ContainerStarted","Data":"c8df550fc3383b27eb726735932eff662da3f5a559447efa113ec1bca1c2c964"} Oct 30 18:45:51.800670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.800644 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:51.845499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:51.845445 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" podStartSLOduration=1.8454302139999998 podStartE2EDuration="1.845430214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:51.844981958 +0000 UTC m=+2850.894026871" watchObservedRunningTime="2026-10-30 18:45:51.845430214 +0000 UTC m=+2850.894475124" Oct 30 18:45:52.115381 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:52.115353 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd240896_9e6b_4cf3_a907_e6d391aa3aa0.slice/crio-efbd1ea4123797f7052c963b4840c73cbe654495ce58d5b80d076b8f182f87d9 WatchSource:0}: Error finding container efbd1ea4123797f7052c963b4840c73cbe654495ce58d5b80d076b8f182f87d9: Status 404 returned error can't find the container with id efbd1ea4123797f7052c963b4840c73cbe654495ce58d5b80d076b8f182f87d9 Oct 30 18:45:52.116422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:52.116392 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp"] Oct 30 18:45:52.616365 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:52.616330 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:45:52.778140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:52.778109 4862 generic.go:356] "Generic (PLEG): container finished" podID="4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3" containerID="8accace7a2770a93927b3de3256dc5fbc751d206e75e8aa65a1ec4aba8a3668c" exitCode=0 Oct 30 18:45:52.778575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:52.778192 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" event={"ID":"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3","Type":"ContainerDied","Data":"8accace7a2770a93927b3de3256dc5fbc751d206e75e8aa65a1ec4aba8a3668c"} Oct 30 18:45:52.779921 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:52.779900 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" event={"ID":"bd240896-9e6b-4cf3-a907-e6d391aa3aa0","Type":"ContainerStarted","Data":"fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480"} Oct 30 18:45:52.780043 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:52.779936 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" event={"ID":"bd240896-9e6b-4cf3-a907-e6d391aa3aa0","Type":"ContainerStarted","Data":"efbd1ea4123797f7052c963b4840c73cbe654495ce58d5b80d076b8f182f87d9"} Oct 30 18:45:52.780043 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:52.779975 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:45:52.858528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:52.858484 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" podStartSLOduration=1.858470578 podStartE2EDuration="1.858470578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:52.856845117 +0000 UTC m=+2851.905890030" watchObservedRunningTime="2026-10-30 18:45:52.858470578 +0000 UTC m=+2851.907515491" Oct 30 18:45:53.939098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:53.939069 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:54.015943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.015914 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-xks-ca\") pod \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " Oct 30 18:45:54.016132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.016009 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-openshift-ca\") pod \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " Oct 30 18:45:54.016132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.016112 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-kube-api-access-8994c\" (UniqueName: \"kubernetes.io/projected/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-kube-api-access-8994c\") pod \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\" (UID: \"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3\") " Oct 30 18:45:54.016415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.016386 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-openshift-ca" pod "4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3" (UID: "4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:54.016782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.016758 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:54.018420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.018394 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-xks-ca" pod "4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3" (UID: "4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:54.018677 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.018655 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-kube-api-access-8994c" pod "4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3" (UID: "4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3"). InnerVolumeSpecName "kube-api-access-8994c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:54.118662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.118628 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:54.118662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.118654 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8994c\" (UniqueName: \"kubernetes.io/projected/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3-kube-api-access-8994c\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:54.796098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.796065 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" event={"ID":"4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3","Type":"ContainerDied","Data":"c8df550fc3383b27eb726735932eff662da3f5a559447efa113ec1bca1c2c964"} Oct 30 18:45:54.796098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.796080 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6" Oct 30 18:45:54.796362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.796112 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8df550fc3383b27eb726735932eff662da3f5a559447efa113ec1bca1c2c964" Oct 30 18:45:54.944124 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:54.944089 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:45:55.694825 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:55.694796 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:45:55.697527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:55.697489 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" podUID="74c01c2c-588b-4480-9e22-c3a71a58ea3c" containerName="payload-processing" containerID="cri-o://6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db" gracePeriod=25 Oct 30 18:45:55.790440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:55.790391 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs"] Oct 30 18:45:56.021222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.021132 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd"] Oct 30 18:45:56.032327 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.028936 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:45:56.051047 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.051019 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn"] Oct 30 18:45:56.052049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.052022 4862 cpu_manager.go:414] "containerMap: removing container" podUID="4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3" containerName="revoke-keys" Oct 30 18:45:56.052049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.052048 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3" containerName="revoke-keys" Oct 30 18:45:56.052249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.052230 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3" containerName="revoke-keys" Oct 30 18:45:56.057448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.056861 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.094432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.094346 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn"] Oct 30 18:45:56.128718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.128689 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:56.192289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.190220 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.192289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.190293 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.192289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.190435 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.192289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.190529 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wrw\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-kube-api-access-r6wrw\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.291678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.291605 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-kube-api-access-stq8h\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-kube-api-access-stq8h\") pod \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " Oct 30 18:45:56.291678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.291652 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-plugins-config-volume\") pod \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " Oct 30 18:45:56.291678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.291680 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-provider-credentials\") pod \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " Oct 30 18:45:56.291978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.291736 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-routing-overlay-volume\") pod \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\" (UID: \"74c01c2c-588b-4480-9e22-c3a71a58ea3c\") " Oct 30 18:45:56.291978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.291892 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wrw\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-kube-api-access-r6wrw\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.292127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.292105 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-plugins-config-volume" pod "74c01c2c-588b-4480-9e22-c3a71a58ea3c" (UID: "74c01c2c-588b-4480-9e22-c3a71a58ea3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:56.292179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.292148 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.292228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.292179 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.292228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.292151 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-routing-overlay-volume" pod "74c01c2c-588b-4480-9e22-c3a71a58ea3c" (UID: "74c01c2c-588b-4480-9e22-c3a71a58ea3c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:56.292351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.292298 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.292421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.292408 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:56.292521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.292506 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74c01c2c-588b-4480-9e22-c3a71a58ea3c-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:56.292933 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.292911 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.293340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.293312 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.294239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.294213 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-kube-api-access-stq8h" pod "74c01c2c-588b-4480-9e22-c3a71a58ea3c" (UID: "74c01c2c-588b-4480-9e22-c3a71a58ea3c"). InnerVolumeSpecName "kube-api-access-stq8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:56.294645 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.294625 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-provider-credentials" pod "74c01c2c-588b-4480-9e22-c3a71a58ea3c" (UID: "74c01c2c-588b-4480-9e22-c3a71a58ea3c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:56.294989 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.294971 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.312111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.312081 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wrw\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-kube-api-access-r6wrw\") pod \"payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.393853 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.393821 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stq8h\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-kube-api-access-stq8h\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:56.393853 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.393850 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74c01c2c-588b-4480-9e22-c3a71a58ea3c-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:45:56.396780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.396747 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2"] Oct 30 18:45:56.397980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.397942 4862 cpu_manager.go:414] "containerMap: removing container" podUID="74c01c2c-588b-4480-9e22-c3a71a58ea3c" containerName="payload-processing" Oct 30 18:45:56.397980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.397968 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c01c2c-588b-4480-9e22-c3a71a58ea3c" containerName="payload-processing" Oct 30 18:45:56.398218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.398198 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c01c2c-588b-4480-9e22-c3a71a58ea3c" containerName="payload-processing" Oct 30 18:45:56.424923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.423662 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2"] Oct 30 18:45:56.424923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.423792 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.426710 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.425642 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:56.441594 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.441537 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:56.441716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.441700 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:45:56.495866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.495472 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2819b79e-rbdb2\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.495866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.495528 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2819b79e-rbdb2\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.495866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.495655 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnts\" (UniqueName: \"kubernetes.io/projected/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-kube-api-access-tdnts\") pod \"maas-api-revoke-sub-2819b79e-rbdb2\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.596790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.596758 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2819b79e-rbdb2\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.596974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.596805 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2819b79e-rbdb2\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.596974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.596892 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnts\" (UniqueName: \"kubernetes.io/projected/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-kube-api-access-tdnts\") pod \"maas-api-revoke-sub-2819b79e-rbdb2\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.597560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.597530 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2819b79e-rbdb2\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.599341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.599312 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2819b79e-rbdb2\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.615897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.615878 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnts\" (UniqueName: \"kubernetes.io/projected/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-kube-api-access-tdnts\") pod \"maas-api-revoke-sub-2819b79e-rbdb2\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.741289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.741234 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn"] Oct 30 18:45:56.741764 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:56.741738 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad763fe5_2f17_4fbc_8890_de1a2bf24cac.slice/crio-f1efcd4122d74c5cbe2c8eacc9c25f9689fe29de5449ba9a55a666dabc3f3623 WatchSource:0}: Error finding container f1efcd4122d74c5cbe2c8eacc9c25f9689fe29de5449ba9a55a666dabc3f3623: Status 404 returned error can't find the container with id f1efcd4122d74c5cbe2c8eacc9c25f9689fe29de5449ba9a55a666dabc3f3623 Oct 30 18:45:56.754772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.754752 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:45:56.833310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.829487 4862 generic.go:356] "Generic (PLEG): container finished" podID="74c01c2c-588b-4480-9e22-c3a71a58ea3c" containerID="6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db" exitCode=0 Oct 30 18:45:56.833310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.829636 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" event={"ID":"74c01c2c-588b-4480-9e22-c3a71a58ea3c","Type":"ContainerDied","Data":"6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db"} Oct 30 18:45:56.833310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.829688 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" event={"ID":"74c01c2c-588b-4480-9e22-c3a71a58ea3c","Type":"ContainerDied","Data":"cde60928bdfc3b360056c11a28f57d928072280c9836a21af7bc7cf42fcca6ea"} Oct 30 18:45:56.833310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.829718 4862 scope.go:122] "RemoveContainer" containerID="6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db" Oct 30 18:45:56.833310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.829896 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst" Oct 30 18:45:56.836230 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.836202 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" event={"ID":"ad763fe5-2f17-4fbc-8890-de1a2bf24cac","Type":"ContainerStarted","Data":"f1efcd4122d74c5cbe2c8eacc9c25f9689fe29de5449ba9a55a666dabc3f3623"} Oct 30 18:45:56.850400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.850373 4862 scope.go:122] "RemoveContainer" containerID="6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db" Oct 30 18:45:56.850728 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:56.850699 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db\": container with ID starting with 6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db not found: ID does not exist" containerID="6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db" Oct 30 18:45:56.850819 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.850740 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db"} err="failed to get container status \"6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db\": rpc error: code = NotFound desc = could not find container \"6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db\": container with ID starting with 6cbddbf106b1c198712f0d5e6a654573667435777f426355994688cac22dc5db not found: ID does not exist" Oct 30 18:45:56.889298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.888791 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst"] Oct 30 18:45:56.895545 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:56.895523 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-69f8jmnst"] Oct 30 18:45:57.064741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:57.064711 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c01c2c-588b-4480-9e22-c3a71a58ea3c" path="/var/lib/kubelet/pods/74c01c2c-588b-4480-9e22-c3a71a58ea3c/volumes" Oct 30 18:45:57.065892 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:45:57.065862 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf54a9fa_e594_47a5_b78b_bfa91fe6dfc6.slice/crio-4284be54e155cda467dd2c28550bdc7a78c41096b16fde79908e4eb4b820d532 WatchSource:0}: Error finding container 4284be54e155cda467dd2c28550bdc7a78c41096b16fde79908e4eb4b820d532: Status 404 returned error can't find the container with id 4284be54e155cda467dd2c28550bdc7a78c41096b16fde79908e4eb4b820d532 Oct 30 18:45:57.066232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:57.066213 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2"] Oct 30 18:45:57.844791 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:57.844749 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" event={"ID":"ad763fe5-2f17-4fbc-8890-de1a2bf24cac","Type":"ContainerStarted","Data":"f12e93b2fabffffc7da6fc6d618c961b88f42520763968d1a437211d816f9889"} Oct 30 18:45:57.844964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:57.844816 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:45:57.846407 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:57.846380 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" event={"ID":"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6","Type":"ContainerStarted","Data":"71ff4ea914340138f899d2ee08644d0764e8c13568c18ba2f60bc7f7a39cc54f"} Oct 30 18:45:57.846530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:57.846424 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" event={"ID":"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6","Type":"ContainerStarted","Data":"4284be54e155cda467dd2c28550bdc7a78c41096b16fde79908e4eb4b820d532"} Oct 30 18:45:57.888544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:57.888500 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" podStartSLOduration=1.8884878729999999 podStartE2EDuration="1.888487873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:57.884895758 +0000 UTC m=+2856.933940671" watchObservedRunningTime="2026-10-30 18:45:57.888487873 +0000 UTC m=+2856.937532786" Oct 30 18:45:57.926178 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:57.926136 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" podStartSLOduration=1.926124088 podStartE2EDuration="1.926124088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:57.92299874 +0000 UTC m=+2856.972043656" watchObservedRunningTime="2026-10-30 18:45:57.926124088 +0000 UTC m=+2856.975169044" Oct 30 18:45:58.856163 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:58.856125 4862 generic.go:356] "Generic (PLEG): container finished" podID="df54a9fa-e594-47a5-b78b-bfa91fe6dfc6" containerID="71ff4ea914340138f899d2ee08644d0764e8c13568c18ba2f60bc7f7a39cc54f" exitCode=0 Oct 30 18:45:58.856576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:45:58.856213 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" event={"ID":"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6","Type":"ContainerDied","Data":"71ff4ea914340138f899d2ee08644d0764e8c13568c18ba2f60bc7f7a39cc54f"} Oct 30 18:45:59.981710 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:45:59.981666 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache]" Oct 30 18:46:00.031429 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.031402 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:46:00.147120 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.143701 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-openshift-ca\") pod \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " Oct 30 18:46:00.147120 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.143800 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-xks-ca\") pod \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " Oct 30 18:46:00.147120 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.143851 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-kube-api-access-tdnts\" (UniqueName: \"kubernetes.io/projected/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-kube-api-access-tdnts\") pod \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\" (UID: \"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6\") " Oct 30 18:46:00.150992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.150955 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-openshift-ca" pod "df54a9fa-e594-47a5-b78b-bfa91fe6dfc6" (UID: "df54a9fa-e594-47a5-b78b-bfa91fe6dfc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:00.154422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.154146 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-kube-api-access-tdnts" pod "df54a9fa-e594-47a5-b78b-bfa91fe6dfc6" (UID: "df54a9fa-e594-47a5-b78b-bfa91fe6dfc6"). InnerVolumeSpecName "kube-api-access-tdnts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:00.154844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.154817 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-xks-ca" pod "df54a9fa-e594-47a5-b78b-bfa91fe6dfc6" (UID: "df54a9fa-e594-47a5-b78b-bfa91fe6dfc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:00.251391 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.251361 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:00.251391 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.251388 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:00.251391 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.251398 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdnts\" (UniqueName: \"kubernetes.io/projected/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6-kube-api-access-tdnts\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:00.791473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.791438 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" podUID="06a8a996-6dbd-4a2e-a63c-88421176c7f9" containerName="payload-processing" containerID="cri-o://ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492" gracePeriod=25 Oct 30 18:46:00.870761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.870735 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" event={"ID":"df54a9fa-e594-47a5-b78b-bfa91fe6dfc6","Type":"ContainerDied","Data":"4284be54e155cda467dd2c28550bdc7a78c41096b16fde79908e4eb4b820d532"} Oct 30 18:46:00.870874 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.870783 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4284be54e155cda467dd2c28550bdc7a78c41096b16fde79908e4eb4b820d532" Oct 30 18:46:00.870874 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:00.870782 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:46:01.025639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.025604 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" podUID="f5ab924d-c201-4f17-9f90-82845d9ca8d5" containerName="payload-processing" containerID="cri-o://bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739" gracePeriod=25 Oct 30 18:46:01.124083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.124059 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:46:01.261561 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.261528 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-kube-api-access-sgd59\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-kube-api-access-sgd59\") pod \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " Oct 30 18:46:01.261735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.261630 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-provider-credentials\") pod \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " Oct 30 18:46:01.261735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.261688 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-plugins-config-volume\") pod \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " Oct 30 18:46:01.261872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.261854 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-routing-overlay-volume\") pod \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\" (UID: \"06a8a996-6dbd-4a2e-a63c-88421176c7f9\") " Oct 30 18:46:01.262207 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.262137 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-plugins-config-volume" pod "06a8a996-6dbd-4a2e-a63c-88421176c7f9" (UID: "06a8a996-6dbd-4a2e-a63c-88421176c7f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:01.262207 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.262162 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-routing-overlay-volume" pod "06a8a996-6dbd-4a2e-a63c-88421176c7f9" (UID: "06a8a996-6dbd-4a2e-a63c-88421176c7f9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:01.262523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.262395 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.262523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.262417 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a8a996-6dbd-4a2e-a63c-88421176c7f9-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.264117 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.264087 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-provider-credentials" pod "06a8a996-6dbd-4a2e-a63c-88421176c7f9" (UID: "06a8a996-6dbd-4a2e-a63c-88421176c7f9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:01.264228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.264139 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-kube-api-access-sgd59" pod "06a8a996-6dbd-4a2e-a63c-88421176c7f9" (UID: "06a8a996-6dbd-4a2e-a63c-88421176c7f9"). InnerVolumeSpecName "kube-api-access-sgd59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:01.298110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.298060 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:46:01.363720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.363691 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.363720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.363716 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgd59\" (UniqueName: \"kubernetes.io/projected/06a8a996-6dbd-4a2e-a63c-88421176c7f9-kube-api-access-sgd59\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.464683 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.464653 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-plugins-config-volume\") pod \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " Oct 30 18:46:01.464851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.464718 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-provider-credentials\") pod \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " Oct 30 18:46:01.464851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.464793 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-kube-api-access-h66cn\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-kube-api-access-h66cn\") pod \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " Oct 30 18:46:01.464851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.464841 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-routing-overlay-volume\") pod \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\" (UID: \"f5ab924d-c201-4f17-9f90-82845d9ca8d5\") " Oct 30 18:46:01.465178 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.465145 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-plugins-config-volume" pod "f5ab924d-c201-4f17-9f90-82845d9ca8d5" (UID: "f5ab924d-c201-4f17-9f90-82845d9ca8d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:01.465342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.465251 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-routing-overlay-volume" pod "f5ab924d-c201-4f17-9f90-82845d9ca8d5" (UID: "f5ab924d-c201-4f17-9f90-82845d9ca8d5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:01.465393 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.465349 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.466931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.466912 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-provider-credentials" pod "f5ab924d-c201-4f17-9f90-82845d9ca8d5" (UID: "f5ab924d-c201-4f17-9f90-82845d9ca8d5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:01.466988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.466970 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-kube-api-access-h66cn" pod "f5ab924d-c201-4f17-9f90-82845d9ca8d5" (UID: "f5ab924d-c201-4f17-9f90-82845d9ca8d5"). InnerVolumeSpecName "kube-api-access-h66cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:01.566157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.566083 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.566157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.566110 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h66cn\" (UniqueName: \"kubernetes.io/projected/f5ab924d-c201-4f17-9f90-82845d9ca8d5-kube-api-access-h66cn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.566157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.566122 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5ab924d-c201-4f17-9f90-82845d9ca8d5-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.878398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.878308 4862 generic.go:356] "Generic (PLEG): container finished" podID="06a8a996-6dbd-4a2e-a63c-88421176c7f9" containerID="ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492" exitCode=0 Oct 30 18:46:01.878398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.878360 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" Oct 30 18:46:01.878398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.878384 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" event={"ID":"06a8a996-6dbd-4a2e-a63c-88421176c7f9","Type":"ContainerDied","Data":"ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492"} Oct 30 18:46:01.878666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.878434 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs" event={"ID":"06a8a996-6dbd-4a2e-a63c-88421176c7f9","Type":"ContainerDied","Data":"42501e9996c38dff4b04a5919ffd540b5cb951d31ff9c7e083dcfedbede77b90"} Oct 30 18:46:01.878666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.878457 4862 scope.go:122] "RemoveContainer" containerID="ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492" Oct 30 18:46:01.879938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.879913 4862 generic.go:356] "Generic (PLEG): container finished" podID="f5ab924d-c201-4f17-9f90-82845d9ca8d5" containerID="bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739" exitCode=0 Oct 30 18:46:01.880043 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.879946 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" event={"ID":"f5ab924d-c201-4f17-9f90-82845d9ca8d5","Type":"ContainerDied","Data":"bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739"} Oct 30 18:46:01.880043 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.879993 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" event={"ID":"f5ab924d-c201-4f17-9f90-82845d9ca8d5","Type":"ContainerDied","Data":"ce317f4b3c922615a7e26fb5f6d5135d146906ee604c0c316237747ee5130768"} Oct 30 18:46:01.880219 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.880199 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd" Oct 30 18:46:01.903211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.902702 4862 scope.go:122] "RemoveContainer" containerID="ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492" Oct 30 18:46:01.903211 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:01.903072 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492\": container with ID starting with ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492 not found: ID does not exist" containerID="ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492" Oct 30 18:46:01.903211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.903096 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492"} err="failed to get container status \"ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492\": rpc error: code = NotFound desc = could not find container \"ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492\": container with ID starting with ec8d4144b3ca98e0a2666f4e0c29330587d92a0636309d41a94291ac6d57b492 not found: ID does not exist" Oct 30 18:46:01.903211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.903114 4862 scope.go:122] "RemoveContainer" containerID="bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739" Oct 30 18:46:01.932592 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.931855 4862 scope.go:122] "RemoveContainer" containerID="bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739" Oct 30 18:46:01.932592 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:01.932315 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739\": container with ID starting with bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739 not found: ID does not exist" containerID="bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739" Oct 30 18:46:01.932592 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.932350 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739"} err="failed to get container status \"bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739\": rpc error: code = NotFound desc = could not find container \"bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739\": container with ID starting with bac06a7197fdb87fdef673cc52a91f4b167006045ff3e8712edb336bbb597739 not found: ID does not exist" Oct 30 18:46:01.981044 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.981014 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd"] Oct 30 18:46:01.995968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:01.995935 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-78f74crnd"] Oct 30 18:46:02.045113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.045083 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs"] Oct 30 18:46:02.050341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.050313 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-85c4lz5vs"] Oct 30 18:46:02.449083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449054 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv"] Oct 30 18:46:02.449669 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449654 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f5ab924d-c201-4f17-9f90-82845d9ca8d5" containerName="payload-processing" Oct 30 18:46:02.449714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449671 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ab924d-c201-4f17-9f90-82845d9ca8d5" containerName="payload-processing" Oct 30 18:46:02.449714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449688 4862 cpu_manager.go:414] "containerMap: removing container" podUID="06a8a996-6dbd-4a2e-a63c-88421176c7f9" containerName="payload-processing" Oct 30 18:46:02.449714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449693 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a8a996-6dbd-4a2e-a63c-88421176c7f9" containerName="payload-processing" Oct 30 18:46:02.449714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449704 4862 cpu_manager.go:414] "containerMap: removing container" podUID="df54a9fa-e594-47a5-b78b-bfa91fe6dfc6" containerName="revoke-keys" Oct 30 18:46:02.449714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449709 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df54a9fa-e594-47a5-b78b-bfa91fe6dfc6" containerName="revoke-keys" Oct 30 18:46:02.449872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449837 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ab924d-c201-4f17-9f90-82845d9ca8d5" containerName="payload-processing" Oct 30 18:46:02.449872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449848 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a8a996-6dbd-4a2e-a63c-88421176c7f9" containerName="payload-processing" Oct 30 18:46:02.449872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.449855 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="df54a9fa-e594-47a5-b78b-bfa91fe6dfc6" containerName="revoke-keys" Oct 30 18:46:02.454651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.454633 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.473275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.473237 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv"] Oct 30 18:46:02.599826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.586641 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2g2\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-kube-api-access-jt2g2\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.599826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.586893 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.599826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.586951 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.599826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.586984 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.687803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.687775 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.687985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.687823 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.687985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.687870 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.687985 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.687941 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2g2\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-kube-api-access-jt2g2\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.688637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.688617 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.688686 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.688643 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.690358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.690341 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.708671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.708615 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2g2\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-kube-api-access-jt2g2\") pod \"payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.747813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.747787 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf"] Oct 30 18:46:02.752005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.751990 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:02.767862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.767840 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:02.768521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.768501 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:46:02.768818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.768795 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:46:02.770614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.770593 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf"] Oct 30 18:46:02.890388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.890361 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97dec1d2-h6gdf\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:02.890573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.890518 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97dec1d2-h6gdf\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:02.890573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.890552 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jv5\" (UniqueName: \"kubernetes.io/projected/35a794af-f0fa-42ce-886c-4eb4be691cda-kube-api-access-n6jv5\") pod \"maas-api-revoke-sub-97dec1d2-h6gdf\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:02.992085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.992010 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97dec1d2-h6gdf\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:02.992085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.992055 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jv5\" (UniqueName: \"kubernetes.io/projected/35a794af-f0fa-42ce-886c-4eb4be691cda-kube-api-access-n6jv5\") pod \"maas-api-revoke-sub-97dec1d2-h6gdf\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:02.992342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.992173 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97dec1d2-h6gdf\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:02.992760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.992736 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97dec1d2-h6gdf\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:02.994685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:02.994665 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97dec1d2-h6gdf\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:03.011699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.011665 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jv5\" (UniqueName: \"kubernetes.io/projected/35a794af-f0fa-42ce-886c-4eb4be691cda-kube-api-access-n6jv5\") pod \"maas-api-revoke-sub-97dec1d2-h6gdf\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:03.067486 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.067413 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:03.071495 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.071469 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a8a996-6dbd-4a2e-a63c-88421176c7f9" path="/var/lib/kubelet/pods/06a8a996-6dbd-4a2e-a63c-88421176c7f9/volumes" Oct 30 18:46:03.071912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.071897 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ab924d-c201-4f17-9f90-82845d9ca8d5" path="/var/lib/kubelet/pods/f5ab924d-c201-4f17-9f90-82845d9ca8d5/volumes" Oct 30 18:46:03.081861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.081839 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv"] Oct 30 18:46:03.084563 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:46:03.084542 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf63eab6_6de1_472a_8198_9dd5d5c5a59a.slice/crio-b3f825f4e25f7bcdc9e66895ee55c1d086765722591ef6b0ed1b25978aa8fd8a WatchSource:0}: Error finding container b3f825f4e25f7bcdc9e66895ee55c1d086765722591ef6b0ed1b25978aa8fd8a: Status 404 returned error can't find the container with id b3f825f4e25f7bcdc9e66895ee55c1d086765722591ef6b0ed1b25978aa8fd8a Oct 30 18:46:03.379445 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:46:03.379416 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a794af_f0fa_42ce_886c_4eb4be691cda.slice/crio-b3cdf0e2a855131e63ebb3431d77765f29a546ca3c7afbca977802534ad10e93 WatchSource:0}: Error finding container b3cdf0e2a855131e63ebb3431d77765f29a546ca3c7afbca977802534ad10e93: Status 404 returned error can't find the container with id b3cdf0e2a855131e63ebb3431d77765f29a546ca3c7afbca977802534ad10e93 Oct 30 18:46:03.379577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.379439 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf"] Oct 30 18:46:03.790258 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.790233 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:46:03.904070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.903974 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" event={"ID":"cf63eab6-6de1-472a-8198-9dd5d5c5a59a","Type":"ContainerStarted","Data":"b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3"} Oct 30 18:46:03.904070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.904040 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" event={"ID":"cf63eab6-6de1-472a-8198-9dd5d5c5a59a","Type":"ContainerStarted","Data":"b3f825f4e25f7bcdc9e66895ee55c1d086765722591ef6b0ed1b25978aa8fd8a"} Oct 30 18:46:03.904303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.904071 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:03.905638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.905612 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" event={"ID":"35a794af-f0fa-42ce-886c-4eb4be691cda","Type":"ContainerStarted","Data":"b5b21e1a777d4bb0c4743e59d7208a13978df152e80d4697317edde381fafbc6"} Oct 30 18:46:03.905763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.905654 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" event={"ID":"35a794af-f0fa-42ce-886c-4eb4be691cda","Type":"ContainerStarted","Data":"b3cdf0e2a855131e63ebb3431d77765f29a546ca3c7afbca977802534ad10e93"} Oct 30 18:46:03.947448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.947406 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" podStartSLOduration=1.947394474 podStartE2EDuration="1.947394474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:03.946706846 +0000 UTC m=+2862.995751759" watchObservedRunningTime="2026-10-30 18:46:03.947394474 +0000 UTC m=+2862.996439388" Oct 30 18:46:03.985852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:03.985809 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" podStartSLOduration=1.985797585 podStartE2EDuration="1.985797585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:03.984887172 +0000 UTC m=+2863.033932086" watchObservedRunningTime="2026-10-30 18:46:03.985797585 +0000 UTC m=+2863.034842498" Oct 30 18:46:04.913888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:04.913857 4862 generic.go:356] "Generic (PLEG): container finished" podID="35a794af-f0fa-42ce-886c-4eb4be691cda" containerID="b5b21e1a777d4bb0c4743e59d7208a13978df152e80d4697317edde381fafbc6" exitCode=0 Oct 30 18:46:04.914303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:04.913942 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" event={"ID":"35a794af-f0fa-42ce-886c-4eb4be691cda","Type":"ContainerDied","Data":"b5b21e1a777d4bb0c4743e59d7208a13978df152e80d4697317edde381fafbc6"} Oct 30 18:46:06.075628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.075605 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:06.148984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.148952 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-openshift-ca\") pod \"35a794af-f0fa-42ce-886c-4eb4be691cda\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " Oct 30 18:46:06.149155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.149022 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a794af-f0fa-42ce-886c-4eb4be691cda-kube-api-access-n6jv5\" (UniqueName: \"kubernetes.io/projected/35a794af-f0fa-42ce-886c-4eb4be691cda-kube-api-access-n6jv5\") pod \"35a794af-f0fa-42ce-886c-4eb4be691cda\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " Oct 30 18:46:06.149155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.149055 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-xks-ca\") pod \"35a794af-f0fa-42ce-886c-4eb4be691cda\" (UID: \"35a794af-f0fa-42ce-886c-4eb4be691cda\") " Oct 30 18:46:06.149419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.149382 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-openshift-ca" pod "35a794af-f0fa-42ce-886c-4eb4be691cda" (UID: "35a794af-f0fa-42ce-886c-4eb4be691cda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:06.149670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.149650 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:06.151882 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.151859 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a794af-f0fa-42ce-886c-4eb4be691cda-kube-api-access-n6jv5" pod "35a794af-f0fa-42ce-886c-4eb4be691cda" (UID: "35a794af-f0fa-42ce-886c-4eb4be691cda"). InnerVolumeSpecName "kube-api-access-n6jv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:06.151882 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.151867 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-xks-ca" pod "35a794af-f0fa-42ce-886c-4eb4be691cda" (UID: "35a794af-f0fa-42ce-886c-4eb4be691cda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:06.250761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.250687 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jv5\" (UniqueName: \"kubernetes.io/projected/35a794af-f0fa-42ce-886c-4eb4be691cda-kube-api-access-n6jv5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:06.250761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.250716 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35a794af-f0fa-42ce-886c-4eb4be691cda-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:06.929823 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.929796 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" Oct 30 18:46:06.929971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.929822 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf" event={"ID":"35a794af-f0fa-42ce-886c-4eb4be691cda","Type":"ContainerDied","Data":"b3cdf0e2a855131e63ebb3431d77765f29a546ca3c7afbca977802534ad10e93"} Oct 30 18:46:06.929971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:06.929866 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3cdf0e2a855131e63ebb3431d77765f29a546ca3c7afbca977802534ad10e93" Oct 30 18:46:07.695941 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:07.695905 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:46:08.859566 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:08.859536 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:46:08.931188 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:08.931158 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg"] Oct 30 18:46:09.997607 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:09.997570 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache]" Oct 30 18:46:13.931556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:13.931465 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" podUID="ee9baac7-c92c-4564-b098-ac80fc122bf2" containerName="payload-processing" containerID="cri-o://b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43" gracePeriod=25 Oct 30 18:46:14.208971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.208941 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:46:14.344157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.344113 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-plugins-config-volume\") pod \"ee9baac7-c92c-4564-b098-ac80fc122bf2\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " Oct 30 18:46:14.344374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.344187 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-provider-credentials\") pod \"ee9baac7-c92c-4564-b098-ac80fc122bf2\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " Oct 30 18:46:14.344374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.344247 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-routing-overlay-volume\") pod \"ee9baac7-c92c-4564-b098-ac80fc122bf2\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " Oct 30 18:46:14.344374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.344304 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-kube-api-access-ncv69\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-kube-api-access-ncv69\") pod \"ee9baac7-c92c-4564-b098-ac80fc122bf2\" (UID: \"ee9baac7-c92c-4564-b098-ac80fc122bf2\") " Oct 30 18:46:14.344655 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.344626 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-plugins-config-volume" pod "ee9baac7-c92c-4564-b098-ac80fc122bf2" (UID: "ee9baac7-c92c-4564-b098-ac80fc122bf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:14.344799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.344669 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-routing-overlay-volume" pod "ee9baac7-c92c-4564-b098-ac80fc122bf2" (UID: "ee9baac7-c92c-4564-b098-ac80fc122bf2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:14.344920 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.344897 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:14.344999 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.344926 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee9baac7-c92c-4564-b098-ac80fc122bf2-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:14.346713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.346688 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-kube-api-access-ncv69" pod "ee9baac7-c92c-4564-b098-ac80fc122bf2" (UID: "ee9baac7-c92c-4564-b098-ac80fc122bf2"). InnerVolumeSpecName "kube-api-access-ncv69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:14.346980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.346966 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-provider-credentials" pod "ee9baac7-c92c-4564-b098-ac80fc122bf2" (UID: "ee9baac7-c92c-4564-b098-ac80fc122bf2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:14.446239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.446161 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:14.446239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.446190 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncv69\" (UniqueName: \"kubernetes.io/projected/ee9baac7-c92c-4564-b098-ac80fc122bf2-kube-api-access-ncv69\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:14.918182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.918154 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:14.949297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.949245 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h"] Oct 30 18:46:14.952098 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:14.952069 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6babca/agc-cred-86a75f599ff5: secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:14.952098 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:14.952089 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv: secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:14.952281 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:14.952127 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials podName:cf63eab6-6de1-472a-8198-9dd5d5c5a59a nodeName:}" failed. No retries permitted until 2026-10-30 18:46:15.4521131 +0000 UTC m=+2874.501157991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" (UID: "cf63eab6-6de1-472a-8198-9dd5d5c5a59a") : secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:14.952548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.952528 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv"] Oct 30 18:46:14.998822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.998787 4862 generic.go:356] "Generic (PLEG): container finished" podID="ee9baac7-c92c-4564-b098-ac80fc122bf2" containerID="b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43" exitCode=0 Oct 30 18:46:14.998982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.998834 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" Oct 30 18:46:14.998982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.998873 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" event={"ID":"ee9baac7-c92c-4564-b098-ac80fc122bf2","Type":"ContainerDied","Data":"b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43"} Oct 30 18:46:14.998982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.998929 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg" event={"ID":"ee9baac7-c92c-4564-b098-ac80fc122bf2","Type":"ContainerDied","Data":"a7333802dfc5a99eaeaeb348061ba7ae61b13c42736943173e840ba7c1622060"} Oct 30 18:46:14.998982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:14.998960 4862 scope.go:122] "RemoveContainer" containerID="b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43" Oct 30 18:46:15.011230 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:15.011212 4862 scope.go:122] "RemoveContainer" containerID="b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43" Oct 30 18:46:15.011525 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:15.011504 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43\": container with ID starting with b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43 not found: ID does not exist" containerID="b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43" Oct 30 18:46:15.011581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:15.011533 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43"} err="failed to get container status \"b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43\": rpc error: code = NotFound desc = could not find container \"b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43\": container with ID starting with b04b60cd41b9b206497f68d59c4b971b8fbc75a18539927eaf27591ae6230d43 not found: ID does not exist" Oct 30 18:46:15.156382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:15.156354 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg"] Oct 30 18:46:15.175852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:15.175785 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-644cgjdfg"] Oct 30 18:46:15.458438 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:15.458341 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6babca/agc-cred-86a75f599ff5: secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:15.458674 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:15.458653 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv: secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:15.458793 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:15.458761 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials podName:cf63eab6-6de1-472a-8198-9dd5d5c5a59a nodeName:}" failed. No retries permitted until 2026-10-30 18:46:16.458737751 +0000 UTC m=+2875.507782658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" (UID: "cf63eab6-6de1-472a-8198-9dd5d5c5a59a") : secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:16.468914 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:16.468880 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6babca/agc-cred-86a75f599ff5: secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:16.468914 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:16.468905 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv: secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:16.469482 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:16.468978 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials podName:cf63eab6-6de1-472a-8198-9dd5d5c5a59a nodeName:}" failed. No retries permitted until 2026-10-30 18:46:18.468959595 +0000 UTC m=+2877.518004498 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" (UID: "cf63eab6-6de1-472a-8198-9dd5d5c5a59a") : secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:17.063059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:17.063024 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9baac7-c92c-4564-b098-ac80fc122bf2" path="/var/lib/kubelet/pods/ee9baac7-c92c-4564-b098-ac80fc122bf2/volumes" Oct 30 18:46:17.829806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:17.829774 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn"] Oct 30 18:46:18.492625 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:18.492597 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6babca/agc-cred-86a75f599ff5: secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:18.492625 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:18.492619 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv: secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:18.492846 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:18.492677 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials podName:cf63eab6-6de1-472a-8198-9dd5d5c5a59a nodeName:}" failed. No retries permitted until 2026-10-30 18:46:22.492657233 +0000 UTC m=+2881.541702131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" (UID: "cf63eab6-6de1-472a-8198-9dd5d5c5a59a") : secret "agc-cred-86a75f599ff5" not found Oct 30 18:46:19.871952 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:19.871915 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:46:19.950169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:19.950103 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" podUID="6d19e264-9bc0-4640-acda-d9825ee65692" containerName="payload-processing" containerID="cri-o://5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12" gracePeriod=25 Oct 30 18:46:20.000113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.000075 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" podUID="cf63eab6-6de1-472a-8198-9dd5d5c5a59a" containerName="payload-processing" containerID="cri-o://b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3" gracePeriod=25 Oct 30 18:46:20.077338 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:20.077303 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2168db_5894_4b73_ba78_754f1d8863bc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc189cb4_8886_479c_903a_752c5ad19901.slice/crio-d4acf831596bd050512a971a629b2550b76b46d4b7df86053dc12b135fa8d5ef\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb49158_4950_4fcb_9ddc_aaa5005a019f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:46:20.267099 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.267066 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:46:20.304705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.304681 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:20.415762 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.415737 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-routing-overlay-volume\") pod \"6d19e264-9bc0-4640-acda-d9825ee65692\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " Oct 30 18:46:20.415958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.415785 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-kube-api-access-g58sw\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-kube-api-access-g58sw\") pod \"6d19e264-9bc0-4640-acda-d9825ee65692\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " Oct 30 18:46:20.415958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.415813 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials\") pod \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " Oct 30 18:46:20.415958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.415892 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-kube-api-access-jt2g2\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-kube-api-access-jt2g2\") pod \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " Oct 30 18:46:20.415958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.415932 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-routing-overlay-volume\") pod \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " Oct 30 18:46:20.416232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.415957 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-plugins-config-volume\") pod \"6d19e264-9bc0-4640-acda-d9825ee65692\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " Oct 30 18:46:20.416232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.416057 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-plugins-config-volume\") pod \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\" (UID: \"cf63eab6-6de1-472a-8198-9dd5d5c5a59a\") " Oct 30 18:46:20.416232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.416149 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-provider-credentials\") pod \"6d19e264-9bc0-4640-acda-d9825ee65692\" (UID: \"6d19e264-9bc0-4640-acda-d9825ee65692\") " Oct 30 18:46:20.416443 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.416293 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-routing-overlay-volume" pod "6d19e264-9bc0-4640-acda-d9825ee65692" (UID: "6d19e264-9bc0-4640-acda-d9825ee65692"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:20.416443 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.416355 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-routing-overlay-volume" pod "cf63eab6-6de1-472a-8198-9dd5d5c5a59a" (UID: "cf63eab6-6de1-472a-8198-9dd5d5c5a59a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:20.416718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.416466 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-plugins-config-volume" pod "6d19e264-9bc0-4640-acda-d9825ee65692" (UID: "6d19e264-9bc0-4640-acda-d9825ee65692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:20.416813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.416782 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-plugins-config-volume" pod "cf63eab6-6de1-472a-8198-9dd5d5c5a59a" (UID: "cf63eab6-6de1-472a-8198-9dd5d5c5a59a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:20.417002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.416967 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:20.417002 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.416989 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:20.417158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.417003 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d19e264-9bc0-4640-acda-d9825ee65692-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:20.417158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.417018 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:20.418313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.418291 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-kube-api-access-g58sw" pod "6d19e264-9bc0-4640-acda-d9825ee65692" (UID: "6d19e264-9bc0-4640-acda-d9825ee65692"). InnerVolumeSpecName "kube-api-access-g58sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:20.418507 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.418485 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials" pod "cf63eab6-6de1-472a-8198-9dd5d5c5a59a" (UID: "cf63eab6-6de1-472a-8198-9dd5d5c5a59a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:20.418789 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.418764 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-kube-api-access-jt2g2" pod "cf63eab6-6de1-472a-8198-9dd5d5c5a59a" (UID: "cf63eab6-6de1-472a-8198-9dd5d5c5a59a"). InnerVolumeSpecName "kube-api-access-jt2g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:20.418789 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.418783 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-provider-credentials" pod "6d19e264-9bc0-4640-acda-d9825ee65692" (UID: "6d19e264-9bc0-4640-acda-d9825ee65692"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:20.518192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.518164 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:20.518192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.518187 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g58sw\" (UniqueName: \"kubernetes.io/projected/6d19e264-9bc0-4640-acda-d9825ee65692-kube-api-access-g58sw\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:20.518192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.518197 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:20.518396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:20.518205 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt2g2\" (UniqueName: \"kubernetes.io/projected/cf63eab6-6de1-472a-8198-9dd5d5c5a59a-kube-api-access-jt2g2\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:21.045708 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.045676 4862 generic.go:356] "Generic (PLEG): container finished" podID="6d19e264-9bc0-4640-acda-d9825ee65692" containerID="5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12" exitCode=0 Oct 30 18:46:21.046122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.045721 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" Oct 30 18:46:21.046122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.045748 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" event={"ID":"6d19e264-9bc0-4640-acda-d9825ee65692","Type":"ContainerDied","Data":"5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12"} Oct 30 18:46:21.046122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.045818 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h" event={"ID":"6d19e264-9bc0-4640-acda-d9825ee65692","Type":"ContainerDied","Data":"7a699fb6d62c447805daec07e3171a42df8fc703664676c4c6471b195acce670"} Oct 30 18:46:21.046122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.045843 4862 scope.go:122] "RemoveContainer" containerID="5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12" Oct 30 18:46:21.047607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.047583 4862 generic.go:356] "Generic (PLEG): container finished" podID="cf63eab6-6de1-472a-8198-9dd5d5c5a59a" containerID="b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3" exitCode=0 Oct 30 18:46:21.047724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.047626 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" Oct 30 18:46:21.047724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.047670 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" event={"ID":"cf63eab6-6de1-472a-8198-9dd5d5c5a59a","Type":"ContainerDied","Data":"b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3"} Oct 30 18:46:21.047804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.047722 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv" event={"ID":"cf63eab6-6de1-472a-8198-9dd5d5c5a59a","Type":"ContainerDied","Data":"b3f825f4e25f7bcdc9e66895ee55c1d086765722591ef6b0ed1b25978aa8fd8a"} Oct 30 18:46:21.057702 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.057680 4862 scope.go:122] "RemoveContainer" containerID="5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12" Oct 30 18:46:21.057971 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:21.057956 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12\": container with ID starting with 5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12 not found: ID does not exist" containerID="5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12" Oct 30 18:46:21.058051 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.057983 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12"} err="failed to get container status \"5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12\": rpc error: code = NotFound desc = could not find container \"5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12\": container with ID starting with 5999d7ff5be88074fe3ea843df9d0b84ddf386b6b7c589906ac558d338528e12 not found: ID does not exist" Oct 30 18:46:21.058051 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.058004 4862 scope.go:122] "RemoveContainer" containerID="b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3" Oct 30 18:46:21.069081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.069065 4862 scope.go:122] "RemoveContainer" containerID="b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3" Oct 30 18:46:21.069374 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:21.069357 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3\": container with ID starting with b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3 not found: ID does not exist" containerID="b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3" Oct 30 18:46:21.069431 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.069382 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3"} err="failed to get container status \"b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3\": rpc error: code = NotFound desc = could not find container \"b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3\": container with ID starting with b77f23b42fb5c377739bab77bf4101d86940f9173cf8d2bd7a8716f560ff33e3 not found: ID does not exist" Oct 30 18:46:21.101720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.101694 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h"] Oct 30 18:46:21.113615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.113593 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-5997z884h"] Oct 30 18:46:21.141613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.141588 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv"] Oct 30 18:46:21.149279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:21.149243 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6babca/payload-processing-external-model-e2e-extmod-b-6babca-769bxxvjv"] Oct 30 18:46:22.831080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:22.831040 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" podUID="ad763fe5-2f17-4fbc-8890-de1a2bf24cac" containerName="payload-processing" containerID="cri-o://f12e93b2fabffffc7da6fc6d618c961b88f42520763968d1a437211d816f9889" gracePeriod=25 Oct 30 18:46:23.064435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.064406 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d19e264-9bc0-4640-acda-d9825ee65692" path="/var/lib/kubelet/pods/6d19e264-9bc0-4640-acda-d9825ee65692/volumes" Oct 30 18:46:23.065005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.064987 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf63eab6-6de1-472a-8198-9dd5d5c5a59a" path="/var/lib/kubelet/pods/cf63eab6-6de1-472a-8198-9dd5d5c5a59a/volumes" Oct 30 18:46:23.067148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.067126 4862 generic.go:356] "Generic (PLEG): container finished" podID="ad763fe5-2f17-4fbc-8890-de1a2bf24cac" containerID="f12e93b2fabffffc7da6fc6d618c961b88f42520763968d1a437211d816f9889" exitCode=0 Oct 30 18:46:23.067293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.067205 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" event={"ID":"ad763fe5-2f17-4fbc-8890-de1a2bf24cac","Type":"ContainerDied","Data":"f12e93b2fabffffc7da6fc6d618c961b88f42520763968d1a437211d816f9889"} Oct 30 18:46:23.092492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.092440 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:46:23.246642 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.246613 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-plugins-config-volume\") pod \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " Oct 30 18:46:23.246816 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.246651 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-kube-api-access-r6wrw\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-kube-api-access-r6wrw\") pod \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " Oct 30 18:46:23.246816 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.246702 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-provider-credentials\") pod \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " Oct 30 18:46:23.246816 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.246737 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-routing-overlay-volume\") pod \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\" (UID: \"ad763fe5-2f17-4fbc-8890-de1a2bf24cac\") " Oct 30 18:46:23.247129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.247105 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-plugins-config-volume" pod "ad763fe5-2f17-4fbc-8890-de1a2bf24cac" (UID: "ad763fe5-2f17-4fbc-8890-de1a2bf24cac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:23.247207 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.247153 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-routing-overlay-volume" pod "ad763fe5-2f17-4fbc-8890-de1a2bf24cac" (UID: "ad763fe5-2f17-4fbc-8890-de1a2bf24cac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:23.247328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.247310 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:23.247376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.247333 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:23.249022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.249003 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-kube-api-access-r6wrw" pod "ad763fe5-2f17-4fbc-8890-de1a2bf24cac" (UID: "ad763fe5-2f17-4fbc-8890-de1a2bf24cac"). InnerVolumeSpecName "kube-api-access-r6wrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:23.249133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.249022 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-provider-credentials" pod "ad763fe5-2f17-4fbc-8890-de1a2bf24cac" (UID: "ad763fe5-2f17-4fbc-8890-de1a2bf24cac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:23.348772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.348704 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:23.348772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:23.348730 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6wrw\" (UniqueName: \"kubernetes.io/projected/ad763fe5-2f17-4fbc-8890-de1a2bf24cac-kube-api-access-r6wrw\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:24.074901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:24.074824 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" Oct 30 18:46:24.074901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:24.074862 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn" event={"ID":"ad763fe5-2f17-4fbc-8890-de1a2bf24cac","Type":"ContainerDied","Data":"f1efcd4122d74c5cbe2c8eacc9c25f9689fe29de5449ba9a55a666dabc3f3623"} Oct 30 18:46:24.075465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:24.074921 4862 scope.go:122] "RemoveContainer" containerID="f12e93b2fabffffc7da6fc6d618c961b88f42520763968d1a437211d816f9889" Oct 30 18:46:24.127851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:24.127820 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn"] Oct 30 18:46:24.139726 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:24.139702 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8b8e54/payload-processing-external-model-e2e-extmod-b-8b8e54-6679x58rn"] Oct 30 18:46:25.062740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:25.062710 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad763fe5-2f17-4fbc-8890-de1a2bf24cac" path="/var/lib/kubelet/pods/ad763fe5-2f17-4fbc-8890-de1a2bf24cac/volumes" Oct 30 18:46:37.998487 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.998451 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb"] Oct 30 18:46:37.999029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999014 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ee9baac7-c92c-4564-b098-ac80fc122bf2" containerName="payload-processing" Oct 30 18:46:37.999080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999030 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9baac7-c92c-4564-b098-ac80fc122bf2" containerName="payload-processing" Oct 30 18:46:37.999080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999052 4862 cpu_manager.go:414] "containerMap: removing container" podUID="35a794af-f0fa-42ce-886c-4eb4be691cda" containerName="revoke-keys" Oct 30 18:46:37.999080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999057 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a794af-f0fa-42ce-886c-4eb4be691cda" containerName="revoke-keys" Oct 30 18:46:37.999080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999071 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6d19e264-9bc0-4640-acda-d9825ee65692" containerName="payload-processing" Oct 30 18:46:37.999080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999076 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d19e264-9bc0-4640-acda-d9825ee65692" containerName="payload-processing" Oct 30 18:46:37.999248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999086 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cf63eab6-6de1-472a-8198-9dd5d5c5a59a" containerName="payload-processing" Oct 30 18:46:37.999248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999091 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf63eab6-6de1-472a-8198-9dd5d5c5a59a" containerName="payload-processing" Oct 30 18:46:37.999248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999112 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ad763fe5-2f17-4fbc-8890-de1a2bf24cac" containerName="payload-processing" Oct 30 18:46:37.999248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999118 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad763fe5-2f17-4fbc-8890-de1a2bf24cac" containerName="payload-processing" Oct 30 18:46:37.999248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999215 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d19e264-9bc0-4640-acda-d9825ee65692" containerName="payload-processing" Oct 30 18:46:37.999248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999224 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf63eab6-6de1-472a-8198-9dd5d5c5a59a" containerName="payload-processing" Oct 30 18:46:37.999248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999234 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad763fe5-2f17-4fbc-8890-de1a2bf24cac" containerName="payload-processing" Oct 30 18:46:37.999248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999241 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9baac7-c92c-4564-b098-ac80fc122bf2" containerName="payload-processing" Oct 30 18:46:37.999248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:37.999250 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a794af-f0fa-42ce-886c-4eb4be691cda" containerName="revoke-keys" Oct 30 18:46:38.004150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.004130 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.025642 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.025612 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvzc\" (UniqueName: \"kubernetes.io/projected/b26d21de-92bd-4102-9acb-425c3bb2e3bf-kube-api-access-9fvzc\") pod \"maas-api-revoke-sub-f2873db0-bxmvb\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.025756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.025648 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2873db0-bxmvb\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.025756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.025670 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2873db0-bxmvb\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.036297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.036066 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:46:38.036419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.036122 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:46:38.039947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.039926 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb"] Oct 30 18:46:38.127237 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.127209 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvzc\" (UniqueName: \"kubernetes.io/projected/b26d21de-92bd-4102-9acb-425c3bb2e3bf-kube-api-access-9fvzc\") pod \"maas-api-revoke-sub-f2873db0-bxmvb\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.127424 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.127243 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2873db0-bxmvb\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.127424 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.127281 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2873db0-bxmvb\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.127957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.127936 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2873db0-bxmvb\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.129796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.129776 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2873db0-bxmvb\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.147962 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.147937 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvzc\" (UniqueName: \"kubernetes.io/projected/b26d21de-92bd-4102-9acb-425c3bb2e3bf-kube-api-access-9fvzc\") pod \"maas-api-revoke-sub-f2873db0-bxmvb\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.317712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.317679 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:38.638325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.638289 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb"] Oct 30 18:46:38.638843 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:46:38.638817 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26d21de_92bd_4102_9acb_425c3bb2e3bf.slice/crio-5d59708510af900994ec83037236e7b4c3575f84c6588cff1e6a06feda6ec81d WatchSource:0}: Error finding container 5d59708510af900994ec83037236e7b4c3575f84c6588cff1e6a06feda6ec81d: Status 404 returned error can't find the container with id 5d59708510af900994ec83037236e7b4c3575f84c6588cff1e6a06feda6ec81d Oct 30 18:46:38.644979 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.644957 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr"] Oct 30 18:46:38.651347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.651313 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.666109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.666085 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr"] Oct 30 18:46:38.734354 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.734323 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768844fd-cx2wr\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.734501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.734432 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768844fd-cx2wr\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.734557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.734503 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xt6\" (UniqueName: \"kubernetes.io/projected/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-kube-api-access-r9xt6\") pod \"maas-api-revoke-sub-768844fd-cx2wr\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.836249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.836217 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768844fd-cx2wr\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.836467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.836298 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xt6\" (UniqueName: \"kubernetes.io/projected/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-kube-api-access-r9xt6\") pod \"maas-api-revoke-sub-768844fd-cx2wr\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.836467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.836450 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768844fd-cx2wr\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.836898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.836873 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768844fd-cx2wr\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.839073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.839053 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768844fd-cx2wr\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.859834 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.859808 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xt6\" (UniqueName: \"kubernetes.io/projected/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-kube-api-access-r9xt6\") pod \"maas-api-revoke-sub-768844fd-cx2wr\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:38.983305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:38.983284 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:39.176414 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:39.176378 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" event={"ID":"b26d21de-92bd-4102-9acb-425c3bb2e3bf","Type":"ContainerStarted","Data":"85e347463401de5fb543630e356ed745c8e4440a843f4a025d973d396cd50e45"} Oct 30 18:46:39.176799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:39.176449 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" event={"ID":"b26d21de-92bd-4102-9acb-425c3bb2e3bf","Type":"ContainerStarted","Data":"5d59708510af900994ec83037236e7b4c3575f84c6588cff1e6a06feda6ec81d"} Oct 30 18:46:39.218718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:39.218624 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" podStartSLOduration=2.218612358 podStartE2EDuration="2.218612358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:39.217132274 +0000 UTC m=+2898.266177187" watchObservedRunningTime="2026-10-30 18:46:39.218612358 +0000 UTC m=+2898.267657271" Oct 30 18:46:39.290619 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:39.290592 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr"] Oct 30 18:46:39.290852 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:46:39.290825 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d31eb1c_6d5e_43e0_ac39_36a9cc52746a.slice/crio-79915392305b518d9b20a943cc0d95d0adaf2fcac2319a0fde3572b9c52568e4 WatchSource:0}: Error finding container 79915392305b518d9b20a943cc0d95d0adaf2fcac2319a0fde3572b9c52568e4: Status 404 returned error can't find the container with id 79915392305b518d9b20a943cc0d95d0adaf2fcac2319a0fde3572b9c52568e4 Oct 30 18:46:40.192940 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:40.192905 4862 generic.go:356] "Generic (PLEG): container finished" podID="b26d21de-92bd-4102-9acb-425c3bb2e3bf" containerID="85e347463401de5fb543630e356ed745c8e4440a843f4a025d973d396cd50e45" exitCode=0 Oct 30 18:46:40.193454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:40.192986 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" event={"ID":"b26d21de-92bd-4102-9acb-425c3bb2e3bf","Type":"ContainerDied","Data":"85e347463401de5fb543630e356ed745c8e4440a843f4a025d973d396cd50e45"} Oct 30 18:46:40.194990 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:40.194963 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" event={"ID":"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a","Type":"ContainerStarted","Data":"25503e31ecd3bb508e57ba3de821e95cb12941e8063a0ffcb6aaf22433477b05"} Oct 30 18:46:40.195127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:40.195005 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" event={"ID":"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a","Type":"ContainerStarted","Data":"79915392305b518d9b20a943cc0d95d0adaf2fcac2319a0fde3572b9c52568e4"} Oct 30 18:46:40.274309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:40.274240 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" podStartSLOduration=2.27422581 podStartE2EDuration="2.27422581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:40.271768258 +0000 UTC m=+2899.320813183" watchObservedRunningTime="2026-10-30 18:46:40.27422581 +0000 UTC m=+2899.323270790" Oct 30 18:46:41.043598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.043567 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh"] Oct 30 18:46:41.048066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.048047 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.065084 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.065063 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh"] Oct 30 18:46:41.073538 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.073510 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"agc-cred-b58a5af19216\"" Oct 30 18:46:41.073794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.073775 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"kube-root-ca.crt\"" Oct 30 18:46:41.073794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.073785 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"payload-processing-external-model-e2e--541faec9-dockercfg-76bvw\"" Oct 30 18:46:41.073962 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.073826 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"routing-overlay\"" Oct 30 18:46:41.075058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.075039 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"openshift-service-ca.crt\"" Oct 30 18:46:41.075155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.075083 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d1a330\"" Oct 30 18:46:41.205892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.205854 4862 generic.go:356] "Generic (PLEG): container finished" podID="7d31eb1c-6d5e-43e0-ac39-36a9cc52746a" containerID="25503e31ecd3bb508e57ba3de821e95cb12941e8063a0ffcb6aaf22433477b05" exitCode=0 Oct 30 18:46:41.206343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.205948 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" event={"ID":"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a","Type":"ContainerDied","Data":"25503e31ecd3bb508e57ba3de821e95cb12941e8063a0ffcb6aaf22433477b05"} Oct 30 18:46:41.209434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.209398 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.209563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.209463 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.209607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.209564 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9rrb\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-kube-api-access-b9rrb\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.209726 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.209700 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.311972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.311937 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.312129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.312012 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.312129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.312068 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9rrb\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-kube-api-access-b9rrb\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.312129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.312127 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.312913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.312890 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.314155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.314127 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.318663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.318640 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.334374 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.334346 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9rrb\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-kube-api-access-b9rrb\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.362680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.362656 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:41.368332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.368310 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:41.514280 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.514231 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26d21de-92bd-4102-9acb-425c3bb2e3bf-kube-api-access-9fvzc\" (UniqueName: \"kubernetes.io/projected/b26d21de-92bd-4102-9acb-425c3bb2e3bf-kube-api-access-9fvzc\") pod \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " Oct 30 18:46:41.514440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.514315 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-openshift-ca\") pod \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " Oct 30 18:46:41.514534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.514513 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-xks-ca\") pod \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\" (UID: \"b26d21de-92bd-4102-9acb-425c3bb2e3bf\") " Oct 30 18:46:41.514748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.514721 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-openshift-ca" pod "b26d21de-92bd-4102-9acb-425c3bb2e3bf" (UID: "b26d21de-92bd-4102-9acb-425c3bb2e3bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:41.515151 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.515128 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:41.516672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.516638 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26d21de-92bd-4102-9acb-425c3bb2e3bf-kube-api-access-9fvzc" pod "b26d21de-92bd-4102-9acb-425c3bb2e3bf" (UID: "b26d21de-92bd-4102-9acb-425c3bb2e3bf"). InnerVolumeSpecName "kube-api-access-9fvzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:41.516767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.516730 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-xks-ca" pod "b26d21de-92bd-4102-9acb-425c3bb2e3bf" (UID: "b26d21de-92bd-4102-9acb-425c3bb2e3bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:41.616325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.616217 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26d21de-92bd-4102-9acb-425c3bb2e3bf-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:41.616325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.616253 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fvzc\" (UniqueName: \"kubernetes.io/projected/b26d21de-92bd-4102-9acb-425c3bb2e3bf-kube-api-access-9fvzc\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:41.626592 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.626558 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9"] Oct 30 18:46:41.627448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.627429 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b26d21de-92bd-4102-9acb-425c3bb2e3bf" containerName="revoke-keys" Oct 30 18:46:41.627528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.627450 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26d21de-92bd-4102-9acb-425c3bb2e3bf" containerName="revoke-keys" Oct 30 18:46:41.627673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.627646 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26d21de-92bd-4102-9acb-425c3bb2e3bf" containerName="revoke-keys" Oct 30 18:46:41.634783 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.634759 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.650000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.649976 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d1a330\"/\"agc-cred-889ad36a07a2\"" Oct 30 18:46:41.651389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.651361 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9"] Oct 30 18:46:41.691936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.691898 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh"] Oct 30 18:46:41.699241 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:46:41.699214 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13679341_8b66_40c7_a2bf_b75176f31666.slice/crio-0f39b98be36644895210232085189f9d323e8f65579962a7fb028aa1e572f7c4 WatchSource:0}: Error finding container 0f39b98be36644895210232085189f9d323e8f65579962a7fb028aa1e572f7c4: Status 404 returned error can't find the container with id 0f39b98be36644895210232085189f9d323e8f65579962a7fb028aa1e572f7c4 Oct 30 18:46:41.717255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.717222 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.717467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.717445 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.717560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.717546 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppxrn\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-kube-api-access-ppxrn\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.717691 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.717662 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.819299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.819247 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppxrn\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-kube-api-access-ppxrn\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.819475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.819359 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.819475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.819436 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.819599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.819549 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.820287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.820232 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.820287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.820255 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.822160 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.822133 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.839074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.839047 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppxrn\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-kube-api-access-ppxrn\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:41.952069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:41.951995 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:42.221338 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.220334 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" Oct 30 18:46:42.221338 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.220321 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb" event={"ID":"b26d21de-92bd-4102-9acb-425c3bb2e3bf","Type":"ContainerDied","Data":"5d59708510af900994ec83037236e7b4c3575f84c6588cff1e6a06feda6ec81d"} Oct 30 18:46:42.221338 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.220473 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d59708510af900994ec83037236e7b4c3575f84c6588cff1e6a06feda6ec81d" Oct 30 18:46:42.222430 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.222401 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" event={"ID":"13679341-8b66-40c7-a2bf-b75176f31666","Type":"ContainerStarted","Data":"bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0"} Oct 30 18:46:42.222580 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.222455 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" event={"ID":"13679341-8b66-40c7-a2bf-b75176f31666","Type":"ContainerStarted","Data":"0f39b98be36644895210232085189f9d323e8f65579962a7fb028aa1e572f7c4"} Oct 30 18:46:42.222912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.222892 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:42.271013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.270982 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9"] Oct 30 18:46:42.271399 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:46:42.271375 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf01d42_b18a_4984_af8a_9c125c388775.slice/crio-3b1fa19b82368339ee3a3d0722a5ea53b9aa3e54e7defad2808899fdc00b7b4b WatchSource:0}: Error finding container 3b1fa19b82368339ee3a3d0722a5ea53b9aa3e54e7defad2808899fdc00b7b4b: Status 404 returned error can't find the container with id 3b1fa19b82368339ee3a3d0722a5ea53b9aa3e54e7defad2808899fdc00b7b4b Oct 30 18:46:42.274431 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.274390 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" podStartSLOduration=1.274373317 podStartE2EDuration="1.274373317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:42.266025514 +0000 UTC m=+2901.315070428" watchObservedRunningTime="2026-10-30 18:46:42.274373317 +0000 UTC m=+2901.323418230" Oct 30 18:46:42.499107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.499073 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:42.631454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.631420 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-xks-ca\") pod \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " Oct 30 18:46:42.631636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.631534 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-openshift-ca\") pod \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " Oct 30 18:46:42.631717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.631651 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-kube-api-access-r9xt6\" (UniqueName: \"kubernetes.io/projected/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-kube-api-access-r9xt6\") pod \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\" (UID: \"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a\") " Oct 30 18:46:42.631917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.631884 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-openshift-ca" pod "7d31eb1c-6d5e-43e0-ac39-36a9cc52746a" (UID: "7d31eb1c-6d5e-43e0-ac39-36a9cc52746a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:42.632187 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.632164 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:42.633883 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.633859 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-xks-ca" pod "7d31eb1c-6d5e-43e0-ac39-36a9cc52746a" (UID: "7d31eb1c-6d5e-43e0-ac39-36a9cc52746a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:42.633972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.633906 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-kube-api-access-r9xt6" pod "7d31eb1c-6d5e-43e0-ac39-36a9cc52746a" (UID: "7d31eb1c-6d5e-43e0-ac39-36a9cc52746a"). InnerVolumeSpecName "kube-api-access-r9xt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:42.733930 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.733847 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9xt6\" (UniqueName: \"kubernetes.io/projected/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-kube-api-access-r9xt6\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:42.733930 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:42.733882 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:43.231602 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:43.231561 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" event={"ID":"8bf01d42-b18a-4984-af8a-9c125c388775","Type":"ContainerStarted","Data":"a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8"} Oct 30 18:46:43.232035 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:43.231634 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:43.232035 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:43.231661 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" event={"ID":"8bf01d42-b18a-4984-af8a-9c125c388775","Type":"ContainerStarted","Data":"3b1fa19b82368339ee3a3d0722a5ea53b9aa3e54e7defad2808899fdc00b7b4b"} Oct 30 18:46:43.233074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:43.233052 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" Oct 30 18:46:43.233193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:43.233087 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr" event={"ID":"7d31eb1c-6d5e-43e0-ac39-36a9cc52746a","Type":"ContainerDied","Data":"79915392305b518d9b20a943cc0d95d0adaf2fcac2319a0fde3572b9c52568e4"} Oct 30 18:46:43.233193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:43.233145 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79915392305b518d9b20a943cc0d95d0adaf2fcac2319a0fde3572b9c52568e4" Oct 30 18:46:43.276643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:43.276598 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" podStartSLOduration=2.276585413 podStartE2EDuration="2.276585413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:43.273533308 +0000 UTC m=+2902.322578221" watchObservedRunningTime="2026-10-30 18:46:43.276585413 +0000 UTC m=+2902.325630326" Oct 30 18:46:49.709170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:49.709136 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp"] Oct 30 18:46:49.871558 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:49.871517 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:46:53.034181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.034144 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4"] Oct 30 18:46:53.034855 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.034838 4862 cpu_manager.go:414] "containerMap: removing container" podUID="7d31eb1c-6d5e-43e0-ac39-36a9cc52746a" containerName="revoke-keys" Oct 30 18:46:53.034910 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.034857 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d31eb1c-6d5e-43e0-ac39-36a9cc52746a" containerName="revoke-keys" Oct 30 18:46:53.034992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.034983 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d31eb1c-6d5e-43e0-ac39-36a9cc52746a" containerName="revoke-keys" Oct 30 18:46:53.038611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.038592 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.064740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.064709 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4"] Oct 30 18:46:53.065040 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.065018 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"e2e-cmp-legacy-916f1b-provider-cred\"" Oct 30 18:46:53.065218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.065023 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"openshift-service-ca.crt\"" Oct 30 18:46:53.065376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.065080 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"routing-overlay\"" Oct 30 18:46:53.065474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.065097 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"kube-root-ca.crt\"" Oct 30 18:46:53.065562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.065181 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1c4bcd\"" Oct 30 18:46:53.065641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.065308 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1c4bcd\"/\"payload-processing-external-model-e2e--8a6e7c24-dockercfg-mnxxc\"" Oct 30 18:46:53.081313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.081283 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.081427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.081413 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.081476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.081459 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfv97\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-kube-api-access-qfv97\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.081591 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.081578 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.183112 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.183081 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.183433 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.183409 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.183533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.183480 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfv97\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-kube-api-access-qfv97\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.183595 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.183573 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.184156 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.184128 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.184521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.184497 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.194830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.194804 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.205831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.205810 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfv97\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-kube-api-access-qfv97\") pod \"payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.236456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.236431 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:53.353542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.353511 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:53.668863 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:46:53.668680 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f77ec5_bd2d_4914_91a5_9d7a588771af.slice/crio-45791cfdf2470900d0289190394bfd9c67afdfd4f1e2620add95664b5da8df81 WatchSource:0}: Error finding container 45791cfdf2470900d0289190394bfd9c67afdfd4f1e2620add95664b5da8df81: Status 404 returned error can't find the container with id 45791cfdf2470900d0289190394bfd9c67afdfd4f1e2620add95664b5da8df81 Oct 30 18:46:53.669767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.669744 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4"] Oct 30 18:46:53.981030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.980955 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb"] Oct 30 18:46:53.992168 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:53.992143 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.002980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.002953 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb"] Oct 30 18:46:54.007189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.007168 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-74c552ab55b1\"" Oct 30 18:46:54.012302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.011329 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kg274\"" Oct 30 18:46:54.098546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.098501 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-routing-overlay-volume\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.099070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.098656 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f789m\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-kube-api-access-f789m\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.099070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.098742 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-plugins-config-volume\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.099070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.098793 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-provider-credentials\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.200367 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.200334 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-routing-overlay-volume\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.200539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.200461 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f789m\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-kube-api-access-f789m\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.200641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.200554 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-plugins-config-volume\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.201148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.201120 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-routing-overlay-volume\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.201303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.201283 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-plugins-config-volume\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.201356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.201325 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-provider-credentials\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.203829 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.203803 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-provider-credentials\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.220820 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.220795 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f789m\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-kube-api-access-f789m\") pod \"payload-processing-external-model-7d48d4766c-jkhqb\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.242211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.242167 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:46:54.308330 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.308068 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:54.316542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.316460 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh"] Oct 30 18:46:54.326320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.325493 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" event={"ID":"a5f77ec5-bd2d-4914-91a5-9d7a588771af","Type":"ContainerStarted","Data":"6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b"} Oct 30 18:46:54.326320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.325544 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" event={"ID":"a5f77ec5-bd2d-4914-91a5-9d7a588771af","Type":"ContainerStarted","Data":"45791cfdf2470900d0289190394bfd9c67afdfd4f1e2620add95664b5da8df81"} Oct 30 18:46:54.326320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.325605 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:46:54.401686 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.401631 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" podStartSLOduration=1.4016170589999999 podStartE2EDuration="1.401617059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:54.401029724 +0000 UTC m=+2913.450074649" watchObservedRunningTime="2026-10-30 18:46:54.401617059 +0000 UTC m=+2913.450661972" Oct 30 18:46:54.626755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.626731 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb"] Oct 30 18:46:54.628687 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:46:54.628654 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f99068_6db8_431f_8aa3_578439c536ba.slice/crio-3e53ffd67806022cf030010ce776f174d800ddce5a4121c668a87e63b02784db WatchSource:0}: Error finding container 3e53ffd67806022cf030010ce776f174d800ddce5a4121c668a87e63b02784db: Status 404 returned error can't find the container with id 3e53ffd67806022cf030010ce776f174d800ddce5a4121c668a87e63b02784db Oct 30 18:46:54.710161 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.710132 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" podUID="bd240896-9e6b-4cf3-a907-e6d391aa3aa0" containerName="payload-processing" containerID="cri-o://fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480" gracePeriod=25 Oct 30 18:46:54.951566 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:54.951533 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:46:55.010752 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.010722 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-plugins-config-volume\") pod \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " Oct 30 18:46:55.010935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.010792 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-provider-credentials\") pod \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " Oct 30 18:46:55.010935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.010845 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-kube-api-access-c8r6c\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-kube-api-access-c8r6c\") pod \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " Oct 30 18:46:55.011072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.011018 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-routing-overlay-volume\") pod \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\" (UID: \"bd240896-9e6b-4cf3-a907-e6d391aa3aa0\") " Oct 30 18:46:55.011174 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.011138 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-plugins-config-volume" pod "bd240896-9e6b-4cf3-a907-e6d391aa3aa0" (UID: "bd240896-9e6b-4cf3-a907-e6d391aa3aa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:55.011832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.011797 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:55.011832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.011800 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-routing-overlay-volume" pod "bd240896-9e6b-4cf3-a907-e6d391aa3aa0" (UID: "bd240896-9e6b-4cf3-a907-e6d391aa3aa0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:55.013384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.013359 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-kube-api-access-c8r6c" pod "bd240896-9e6b-4cf3-a907-e6d391aa3aa0" (UID: "bd240896-9e6b-4cf3-a907-e6d391aa3aa0"). InnerVolumeSpecName "kube-api-access-c8r6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:55.013477 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.013431 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-provider-credentials" pod "bd240896-9e6b-4cf3-a907-e6d391aa3aa0" (UID: "bd240896-9e6b-4cf3-a907-e6d391aa3aa0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:55.113336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.113300 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:55.113336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.113331 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8r6c\" (UniqueName: \"kubernetes.io/projected/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-kube-api-access-c8r6c\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:55.113744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.113344 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd240896-9e6b-4cf3-a907-e6d391aa3aa0-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:55.333102 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.333065 4862 generic.go:356] "Generic (PLEG): container finished" podID="bd240896-9e6b-4cf3-a907-e6d391aa3aa0" containerID="fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480" exitCode=0 Oct 30 18:46:55.333300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.333122 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" Oct 30 18:46:55.333300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.333150 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" event={"ID":"bd240896-9e6b-4cf3-a907-e6d391aa3aa0","Type":"ContainerDied","Data":"fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480"} Oct 30 18:46:55.333300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.333207 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp" event={"ID":"bd240896-9e6b-4cf3-a907-e6d391aa3aa0","Type":"ContainerDied","Data":"efbd1ea4123797f7052c963b4840c73cbe654495ce58d5b80d076b8f182f87d9"} Oct 30 18:46:55.333300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.333233 4862 scope.go:122] "RemoveContainer" containerID="fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480" Oct 30 18:46:55.334998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.334976 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" event={"ID":"c0f99068-6db8-431f-8aa3-578439c536ba","Type":"ContainerStarted","Data":"bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03"} Oct 30 18:46:55.335139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.335023 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" event={"ID":"c0f99068-6db8-431f-8aa3-578439c536ba","Type":"ContainerStarted","Data":"3e53ffd67806022cf030010ce776f174d800ddce5a4121c668a87e63b02784db"} Oct 30 18:46:55.335139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.335132 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:46:55.343941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.343926 4862 scope.go:122] "RemoveContainer" containerID="fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480" Oct 30 18:46:55.344218 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:46:55.344201 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480\": container with ID starting with fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480 not found: ID does not exist" containerID="fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480" Oct 30 18:46:55.344294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.344228 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480"} err="failed to get container status \"fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480\": rpc error: code = NotFound desc = could not find container \"fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480\": container with ID starting with fc77593e4d828b249cb46eb2cea2d45fab4f7c4c5416923dd07eb77755aa4480 not found: ID does not exist" Oct 30 18:46:55.386201 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.386180 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp"] Oct 30 18:46:55.398016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.397995 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78bf69cc96-bscgp"] Oct 30 18:46:55.425938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.425899 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" podStartSLOduration=2.425889899 podStartE2EDuration="2.425889899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:55.424538424 +0000 UTC m=+2914.473583336" watchObservedRunningTime="2026-10-30 18:46:55.425889899 +0000 UTC m=+2914.474934826" Oct 30 18:46:55.436577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.436539 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz"] Oct 30 18:46:55.437186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.437170 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bd240896-9e6b-4cf3-a907-e6d391aa3aa0" containerName="payload-processing" Oct 30 18:46:55.437328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.437188 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd240896-9e6b-4cf3-a907-e6d391aa3aa0" containerName="payload-processing" Oct 30 18:46:55.437381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.437336 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd240896-9e6b-4cf3-a907-e6d391aa3aa0" containerName="payload-processing" Oct 30 18:46:55.442172 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.442157 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.467009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.465436 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xfzjq\"" Oct 30 18:46:55.467149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.467130 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz"] Oct 30 18:46:55.519354 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.519326 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.519513 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.519435 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.519513 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.519481 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.519607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.519588 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.519655 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.519605 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.519655 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.519635 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51436817-f5b6-433f-afd3-a472760940a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.519750 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.519737 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51436817-f5b6-433f-afd3-a472760940a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.519820 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.519809 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljssm\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-kube-api-access-ljssm\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.519931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.519910 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.621458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.621421 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51436817-f5b6-433f-afd3-a472760940a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.621611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.621521 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljssm\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-kube-api-access-ljssm\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.621611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.621570 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.621696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.621610 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.621696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.621667 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.621804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.621712 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.621804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.621768 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.621804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.621793 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.621953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.621834 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51436817-f5b6-433f-afd3-a472760940a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.622066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.622030 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.622207 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.622187 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.622296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.622208 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.622508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.622484 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.622612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.622594 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51436817-f5b6-433f-afd3-a472760940a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.624193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.624173 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.624343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.624325 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51436817-f5b6-433f-afd3-a472760940a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.640606 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.640581 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.640720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.640705 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljssm\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-kube-api-access-ljssm\") pod \"maas-default-gateway-openshift-default-76597d6855-7jqjz\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:55.756199 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:55.756126 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:56.074114 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:56.074092 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz"] Oct 30 18:46:56.074424 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:46:56.074394 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51436817_f5b6_433f_afd3_a472760940a6.slice/crio-89040f8e18d45509e8068f300b183611d1ea9bd4ad9b6d866aff53bcdd234558 WatchSource:0}: Error finding container 89040f8e18d45509e8068f300b183611d1ea9bd4ad9b6d866aff53bcdd234558: Status 404 returned error can't find the container with id 89040f8e18d45509e8068f300b183611d1ea9bd4ad9b6d866aff53bcdd234558 Oct 30 18:46:56.077173 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:56.077141 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:46:56.077242 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:56.077203 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:46:56.077242 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:56.077234 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:46:56.345140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:56.345048 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" event={"ID":"51436817-f5b6-433f-afd3-a472760940a6","Type":"ContainerStarted","Data":"300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2"} Oct 30 18:46:56.345140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:56.345110 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" event={"ID":"51436817-f5b6-433f-afd3-a472760940a6","Type":"ContainerStarted","Data":"89040f8e18d45509e8068f300b183611d1ea9bd4ad9b6d866aff53bcdd234558"} Oct 30 18:46:56.389408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:56.389333 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podStartSLOduration=1.389314863 podStartE2EDuration="1.389314863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:56.385640606 +0000 UTC m=+2915.434685511" watchObservedRunningTime="2026-10-30 18:46:56.389314863 +0000 UTC m=+2915.438359778" Oct 30 18:46:56.757245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:56.757167 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:46:57.066740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:57.066599 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd240896-9e6b-4cf3-a907-e6d391aa3aa0" path="/var/lib/kubelet/pods/bd240896-9e6b-4cf3-a907-e6d391aa3aa0/volumes" Oct 30 18:46:57.758548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:57.758507 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7jqjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:46:57.759069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:57.758585 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 18:46:58.757164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:58.757132 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7jqjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:46:58.757395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:58.757199 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 18:46:59.317057 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.317016 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" podUID="13679341-8b66-40c7-a2bf-b75176f31666" containerName="payload-processing" containerID="cri-o://bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0" gracePeriod=25 Oct 30 18:46:59.654730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.654700 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:46:59.756499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.756458 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7jqjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:46:59.756691 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.756529 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 18:46:59.786409 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.786386 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-routing-overlay-volume\") pod \"13679341-8b66-40c7-a2bf-b75176f31666\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " Oct 30 18:46:59.786562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.786440 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-provider-credentials\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-provider-credentials\") pod \"13679341-8b66-40c7-a2bf-b75176f31666\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " Oct 30 18:46:59.786562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.786471 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-kube-api-access-b9rrb\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-kube-api-access-b9rrb\") pod \"13679341-8b66-40c7-a2bf-b75176f31666\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " Oct 30 18:46:59.786562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.786493 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-plugins-config-volume\") pod \"13679341-8b66-40c7-a2bf-b75176f31666\" (UID: \"13679341-8b66-40c7-a2bf-b75176f31666\") " Oct 30 18:46:59.786845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.786812 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-routing-overlay-volume" pod "13679341-8b66-40c7-a2bf-b75176f31666" (UID: "13679341-8b66-40c7-a2bf-b75176f31666"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:59.786969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.786946 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-plugins-config-volume" pod "13679341-8b66-40c7-a2bf-b75176f31666" (UID: "13679341-8b66-40c7-a2bf-b75176f31666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:59.787016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.786965 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:59.788824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.788802 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-provider-credentials" pod "13679341-8b66-40c7-a2bf-b75176f31666" (UID: "13679341-8b66-40c7-a2bf-b75176f31666"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:59.788824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.788814 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-kube-api-access-b9rrb" pod "13679341-8b66-40c7-a2bf-b75176f31666" (UID: "13679341-8b66-40c7-a2bf-b75176f31666"). InnerVolumeSpecName "kube-api-access-b9rrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:59.857856 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.857772 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2"] Oct 30 18:46:59.890305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.889920 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:59.890305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.889944 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9rrb\" (UniqueName: \"kubernetes.io/projected/13679341-8b66-40c7-a2bf-b75176f31666-kube-api-access-b9rrb\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:46:59.890305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:46:59.889954 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13679341-8b66-40c7-a2bf-b75176f31666-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:00.384699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.384176 4862 generic.go:356] "Generic (PLEG): container finished" podID="13679341-8b66-40c7-a2bf-b75176f31666" containerID="bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0" exitCode=0 Oct 30 18:47:00.384699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.384289 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" event={"ID":"13679341-8b66-40c7-a2bf-b75176f31666","Type":"ContainerDied","Data":"bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0"} Oct 30 18:47:00.384699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.384348 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" event={"ID":"13679341-8b66-40c7-a2bf-b75176f31666","Type":"ContainerDied","Data":"0f39b98be36644895210232085189f9d323e8f65579962a7fb028aa1e572f7c4"} Oct 30 18:47:00.384699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.384375 4862 scope.go:122] "RemoveContainer" containerID="bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0" Oct 30 18:47:00.384699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.384507 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh" Oct 30 18:47:00.398275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.398244 4862 scope.go:122] "RemoveContainer" containerID="bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0" Oct 30 18:47:00.398593 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:00.398570 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0\": container with ID starting with bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0 not found: ID does not exist" containerID="bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0" Oct 30 18:47:00.398676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.398603 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0"} err="failed to get container status \"bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0\": rpc error: code = NotFound desc = could not find container \"bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0\": container with ID starting with bf7af861a6fa90974d33d107bb1e6542fe2685240ad90ab532f93a23530200a0 not found: ID does not exist" Oct 30 18:47:00.459840 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.459802 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh"] Oct 30 18:47:00.460098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.460083 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-7964pn2sh"] Oct 30 18:47:00.761331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.760999 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7jqjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:47:00.761331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:00.761059 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 18:47:01.071161 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.071130 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13679341-8b66-40c7-a2bf-b75176f31666" path="/var/lib/kubelet/pods/13679341-8b66-40c7-a2bf-b75176f31666/volumes" Oct 30 18:47:01.560341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.557796 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl"] Oct 30 18:47:01.561918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.561892 4862 cpu_manager.go:414] "containerMap: removing container" podUID="13679341-8b66-40c7-a2bf-b75176f31666" containerName="payload-processing" Oct 30 18:47:01.562089 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.562076 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13679341-8b66-40c7-a2bf-b75176f31666" containerName="payload-processing" Oct 30 18:47:01.562366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.562351 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="13679341-8b66-40c7-a2bf-b75176f31666" containerName="payload-processing" Oct 30 18:47:01.568069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.568052 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl"] Oct 30 18:47:01.568232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.568150 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.585131 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.583423 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"agc-cred-ccddb066b30b\"" Oct 30 18:47:01.595861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.595840 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"payload-processing-external-model-e2e--41b9a229-dockercfg-8jm57\"" Oct 30 18:47:01.652288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.652193 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnng\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-kube-api-access-lbnng\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.652488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.652322 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.652488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.652371 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.652622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.652543 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.759289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.759113 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.759289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.759286 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnng\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-kube-api-access-lbnng\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.759573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.759374 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.759573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.759400 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.759705 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:01.759600 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:47:01.759705 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:01.759617 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl: secret "agc-cred-ccddb066b30b" not found Oct 30 18:47:01.759705 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:01.759691 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials podName:6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5 nodeName:}" failed. No retries permitted until 2026-10-30 18:47:02.259669822 +0000 UTC m=+2921.308714733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" (UID: "6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:47:01.761285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.760707 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.762320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.761743 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:01.762447 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.762343 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7jqjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:47:01.762447 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.762382 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 18:47:01.763614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.763590 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl"] Oct 30 18:47:01.763911 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:01.763893 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lbnng provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" podUID="6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5" Oct 30 18:47:01.788537 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:01.788503 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnng\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-kube-api-access-lbnng\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:02.267385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.267348 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:02.267589 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:02.267498 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:47:02.267589 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:02.267518 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl: secret "agc-cred-ccddb066b30b" not found Oct 30 18:47:02.267589 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:02.267581 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials podName:6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5 nodeName:}" failed. No retries permitted until 2026-10-30 18:47:03.267564417 +0000 UTC m=+2922.316609321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" (UID: "6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:47:02.457731 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.457704 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:02.467294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.467255 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:02.571036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.571005 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-plugins-config-volume\") pod \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " Oct 30 18:47:02.571548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.571101 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-routing-overlay-volume\") pod \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " Oct 30 18:47:02.571548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.571219 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-kube-api-access-lbnng\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-kube-api-access-lbnng\") pod \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " Oct 30 18:47:02.571548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.571498 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-plugins-config-volume" pod "6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5" (UID: "6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:02.571548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.571532 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-routing-overlay-volume" pod "6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5" (UID: "6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:02.571893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.571872 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:02.572156 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.571898 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:02.573529 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.573504 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-kube-api-access-lbnng" pod "6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5" (UID: "6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5"). InnerVolumeSpecName "kube-api-access-lbnng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:02.673387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.673356 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnng\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-kube-api-access-lbnng\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:02.756635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.756600 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7jqjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:47:02.756936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:02.756908 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 18:47:03.280191 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:03.280158 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl\" (UID: \"6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:03.280416 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:03.280354 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2107d4/agc-cred-ccddb066b30b: secret "agc-cred-ccddb066b30b" not found Oct 30 18:47:03.280416 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:03.280373 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl: secret "agc-cred-ccddb066b30b" not found Oct 30 18:47:03.280548 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:03.280434 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials podName:6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5 nodeName:}" failed. No retries permitted until 2026-10-30 18:47:05.280418179 +0000 UTC m=+2924.329463096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" (UID: "6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5") : secret "agc-cred-ccddb066b30b" not found Oct 30 18:47:03.465334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:03.465242 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl" Oct 30 18:47:03.545743 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:03.545712 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl"] Oct 30 18:47:03.550761 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:03.550733 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-676cbdmxl"] Oct 30 18:47:03.685325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:03.685290 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:03.756968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:03.756890 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7jqjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Oct 30 18:47:03.756968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:03.756946 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Oct 30 18:47:04.858919 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:04.858856 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" podUID="7bfe54c8-1ba5-422b-96b0-2f4a846e083e" containerName="payload-processing" containerID="cri-o://537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120" gracePeriod=25 Oct 30 18:47:05.030886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.030853 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:47:05.031538 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.031479 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:47:05.032604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.032580 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:47:05.077293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.073789 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5" path="/var/lib/kubelet/pods/6ebc9b74-b8d6-4305-855a-b5dc9bd5a5f5/volumes" Oct 30 18:47:05.173592 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.173563 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:47:05.304995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.304916 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-provider-credentials\") pod \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " Oct 30 18:47:05.305195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.305019 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-kube-api-access-p5vwq\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-kube-api-access-p5vwq\") pod \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " Oct 30 18:47:05.305195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.305138 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-routing-overlay-volume\") pod \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " Oct 30 18:47:05.305394 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.305366 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-plugins-config-volume\") pod \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\" (UID: \"7bfe54c8-1ba5-422b-96b0-2f4a846e083e\") " Oct 30 18:47:05.305534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.305506 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-routing-overlay-volume" pod "7bfe54c8-1ba5-422b-96b0-2f4a846e083e" (UID: "7bfe54c8-1ba5-422b-96b0-2f4a846e083e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:05.305726 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.305702 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-plugins-config-volume" pod "7bfe54c8-1ba5-422b-96b0-2f4a846e083e" (UID: "7bfe54c8-1ba5-422b-96b0-2f4a846e083e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:05.306110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.306089 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:05.306155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.306119 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:05.307315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.307291 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-kube-api-access-p5vwq" pod "7bfe54c8-1ba5-422b-96b0-2f4a846e083e" (UID: "7bfe54c8-1ba5-422b-96b0-2f4a846e083e"). InnerVolumeSpecName "kube-api-access-p5vwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:05.307411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.307334 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-provider-credentials" pod "7bfe54c8-1ba5-422b-96b0-2f4a846e083e" (UID: "7bfe54c8-1ba5-422b-96b0-2f4a846e083e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:05.338717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.338700 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:47:05.407854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.407752 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:05.408029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.408006 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5vwq\" (UniqueName: \"kubernetes.io/projected/7bfe54c8-1ba5-422b-96b0-2f4a846e083e-kube-api-access-p5vwq\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:05.480771 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.480738 4862 generic.go:356] "Generic (PLEG): container finished" podID="7bfe54c8-1ba5-422b-96b0-2f4a846e083e" containerID="537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120" exitCode=0 Oct 30 18:47:05.480918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.480788 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" Oct 30 18:47:05.480918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.480827 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" event={"ID":"7bfe54c8-1ba5-422b-96b0-2f4a846e083e","Type":"ContainerDied","Data":"537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120"} Oct 30 18:47:05.480918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.480879 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2" event={"ID":"7bfe54c8-1ba5-422b-96b0-2f4a846e083e","Type":"ContainerDied","Data":"eaabd594e44d7c4d161ed1c44e720f5a1a33088226b66bcfa439361126a4bd18"} Oct 30 18:47:05.480918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.480900 4862 scope.go:122] "RemoveContainer" containerID="537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120" Oct 30 18:47:05.496926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.496908 4862 scope.go:122] "RemoveContainer" containerID="537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120" Oct 30 18:47:05.497239 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:05.497220 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120\": container with ID starting with 537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120 not found: ID does not exist" containerID="537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120" Oct 30 18:47:05.497344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.497248 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120"} err="failed to get container status \"537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120\": rpc error: code = NotFound desc = could not find container \"537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120\": container with ID starting with 537dc34688002ffc785b08c56a87221da36adbc45280211634d32da86ff18120 not found: ID does not exist" Oct 30 18:47:05.532701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.532673 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2"] Oct 30 18:47:05.538396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:05.538375 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-5f79kdmh2"] Oct 30 18:47:06.350210 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:06.350180 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:47:06.362044 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:06.362018 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:47:07.106519 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:07.106482 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bfe54c8-1ba5-422b-96b0-2f4a846e083e" path="/var/lib/kubelet/pods/7bfe54c8-1ba5-422b-96b0-2f4a846e083e/volumes" Oct 30 18:47:10.507065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.507028 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l"] Oct 30 18:47:10.507827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.507807 4862 cpu_manager.go:414] "containerMap: removing container" podUID="7bfe54c8-1ba5-422b-96b0-2f4a846e083e" containerName="payload-processing" Oct 30 18:47:10.507827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.507827 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bfe54c8-1ba5-422b-96b0-2f4a846e083e" containerName="payload-processing" Oct 30 18:47:10.507965 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.507941 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bfe54c8-1ba5-422b-96b0-2f4a846e083e" containerName="payload-processing" Oct 30 18:47:10.513189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.513166 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.530917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.530891 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l"] Oct 30 18:47:10.539093 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.539070 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:47:10.539306 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.539082 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:47:10.608807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.608778 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fl7\" (UniqueName: \"kubernetes.io/projected/7e0e9d63-ce79-4090-adf1-454819cc179c-kube-api-access-p9fl7\") pod \"maas-api-revoke-sub-f2e261ba-tmj7l\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.608961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.608899 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2e261ba-tmj7l\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.609036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.609020 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2e261ba-tmj7l\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.710520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.710490 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fl7\" (UniqueName: \"kubernetes.io/projected/7e0e9d63-ce79-4090-adf1-454819cc179c-kube-api-access-p9fl7\") pod \"maas-api-revoke-sub-f2e261ba-tmj7l\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.710675 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.710564 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2e261ba-tmj7l\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.710675 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.710655 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2e261ba-tmj7l\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.711257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.711214 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2e261ba-tmj7l\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.713420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.713395 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2e261ba-tmj7l\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.730241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.730221 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fl7\" (UniqueName: \"kubernetes.io/projected/7e0e9d63-ce79-4090-adf1-454819cc179c-kube-api-access-p9fl7\") pod \"maas-api-revoke-sub-f2e261ba-tmj7l\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:10.831306 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:10.831248 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:11.143344 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:11.143317 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0e9d63_ce79_4090_adf1_454819cc179c.slice/crio-e460784b540d59d499cddc6a99f7cb331e2a32e426056565c98dbc1f039d3869 WatchSource:0}: Error finding container e460784b540d59d499cddc6a99f7cb331e2a32e426056565c98dbc1f039d3869: Status 404 returned error can't find the container with id e460784b540d59d499cddc6a99f7cb331e2a32e426056565c98dbc1f039d3869 Oct 30 18:47:11.143905 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:11.143882 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l"] Oct 30 18:47:11.529196 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:11.529156 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" event={"ID":"7e0e9d63-ce79-4090-adf1-454819cc179c","Type":"ContainerStarted","Data":"4dbbf2d8c58b350a1810ea01f8401eeb885b957ca65dc2e813bbab31628df4e1"} Oct 30 18:47:11.529603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:11.529227 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" event={"ID":"7e0e9d63-ce79-4090-adf1-454819cc179c","Type":"ContainerStarted","Data":"e460784b540d59d499cddc6a99f7cb331e2a32e426056565c98dbc1f039d3869"} Oct 30 18:47:11.568716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:11.568675 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" podStartSLOduration=1.568662229 podStartE2EDuration="1.568662229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:11.568560615 +0000 UTC m=+2930.617605529" watchObservedRunningTime="2026-10-30 18:47:11.568662229 +0000 UTC m=+2930.617707142" Oct 30 18:47:12.537300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:12.537249 4862 generic.go:356] "Generic (PLEG): container finished" podID="7e0e9d63-ce79-4090-adf1-454819cc179c" containerID="4dbbf2d8c58b350a1810ea01f8401eeb885b957ca65dc2e813bbab31628df4e1" exitCode=0 Oct 30 18:47:12.537300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:12.537295 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" event={"ID":"7e0e9d63-ce79-4090-adf1-454819cc179c","Type":"ContainerDied","Data":"4dbbf2d8c58b350a1810ea01f8401eeb885b957ca65dc2e813bbab31628df4e1"} Oct 30 18:47:13.697535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.697513 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:13.751660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.751626 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-xks-ca\") pod \"7e0e9d63-ce79-4090-adf1-454819cc179c\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " Oct 30 18:47:13.751922 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.751855 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-openshift-ca\") pod \"7e0e9d63-ce79-4090-adf1-454819cc179c\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " Oct 30 18:47:13.752255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.752227 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-openshift-ca" pod "7e0e9d63-ce79-4090-adf1-454819cc179c" (UID: "7e0e9d63-ce79-4090-adf1-454819cc179c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:13.752427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.752407 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0e9d63-ce79-4090-adf1-454819cc179c-kube-api-access-p9fl7\" (UniqueName: \"kubernetes.io/projected/7e0e9d63-ce79-4090-adf1-454819cc179c-kube-api-access-p9fl7\") pod \"7e0e9d63-ce79-4090-adf1-454819cc179c\" (UID: \"7e0e9d63-ce79-4090-adf1-454819cc179c\") " Oct 30 18:47:13.753077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.753055 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:13.754482 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.754454 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-xks-ca" pod "7e0e9d63-ce79-4090-adf1-454819cc179c" (UID: "7e0e9d63-ce79-4090-adf1-454819cc179c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:13.755208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.755183 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0e9d63-ce79-4090-adf1-454819cc179c-kube-api-access-p9fl7" pod "7e0e9d63-ce79-4090-adf1-454819cc179c" (UID: "7e0e9d63-ce79-4090-adf1-454819cc179c"). InnerVolumeSpecName "kube-api-access-p9fl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:13.854189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.854120 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9fl7\" (UniqueName: \"kubernetes.io/projected/7e0e9d63-ce79-4090-adf1-454819cc179c-kube-api-access-p9fl7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:13.854189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:13.854150 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0e9d63-ce79-4090-adf1-454819cc179c-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:14.556503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:14.556418 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" event={"ID":"7e0e9d63-ce79-4090-adf1-454819cc179c","Type":"ContainerDied","Data":"e460784b540d59d499cddc6a99f7cb331e2a32e426056565c98dbc1f039d3869"} Oct 30 18:47:14.556503 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:14.556477 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e460784b540d59d499cddc6a99f7cb331e2a32e426056565c98dbc1f039d3869" Oct 30 18:47:14.556822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:14.556796 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l" Oct 30 18:47:18.804017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:18.803982 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb"] Oct 30 18:47:19.872800 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:19.872763 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:47:21.669128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.669084 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h"] Oct 30 18:47:21.669706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.669690 4862 cpu_manager.go:414] "containerMap: removing container" podUID="7e0e9d63-ce79-4090-adf1-454819cc179c" containerName="revoke-keys" Oct 30 18:47:21.669706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.669705 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0e9d63-ce79-4090-adf1-454819cc179c" containerName="revoke-keys" Oct 30 18:47:21.669826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.669813 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0e9d63-ce79-4090-adf1-454819cc179c" containerName="revoke-keys" Oct 30 18:47:21.673318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.673299 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.690949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.690928 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h"] Oct 30 18:47:21.699464 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.699444 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6rk76\"" Oct 30 18:47:21.770866 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.770835 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrj2c\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-kube-api-access-rrj2c\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.770984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.770961 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-routing-overlay-volume\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.771045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.771035 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-provider-credentials\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.771100 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.771054 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-plugins-config-volume\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.872895 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.872862 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-routing-overlay-volume\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.873058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.872973 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-provider-credentials\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.873058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.873007 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-plugins-config-volume\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.873167 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.873137 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrj2c\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-kube-api-access-rrj2c\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.873764 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.873737 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-routing-overlay-volume\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.874157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.874112 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-plugins-config-volume\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.875974 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.875952 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-provider-credentials\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.893077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.893051 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrj2c\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-kube-api-access-rrj2c\") pod \"payload-processing-external-model-7d48d4766c-p8l8h\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:21.987878 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:21.987809 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:22.093872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.093840 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz"] Oct 30 18:47:22.099200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.099175 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.115589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.115562 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:47:22.117317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.116215 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:47:22.117317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.116963 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz"] Oct 30 18:47:22.134902 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.134873 4862 scope.go:122] "RemoveContainer" containerID="76cd77cfc76d6c9e5613ffbba71ec21320cf3db2ac99d35a2bd238d791c416b9" Oct 30 18:47:22.176832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.176807 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c13625d5-ppbwz\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.176962 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.176937 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56gp\" (UniqueName: \"kubernetes.io/projected/26b8a80b-3296-47ff-9301-3a9ad215c6d2-kube-api-access-g56gp\") pod \"maas-api-revoke-sub-c13625d5-ppbwz\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.177476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.177445 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c13625d5-ppbwz\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.279177 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.279119 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c13625d5-ppbwz\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.279288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.279193 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c13625d5-ppbwz\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.279334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.279256 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56gp\" (UniqueName: \"kubernetes.io/projected/26b8a80b-3296-47ff-9301-3a9ad215c6d2-kube-api-access-g56gp\") pod \"maas-api-revoke-sub-c13625d5-ppbwz\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.279870 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.279837 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c13625d5-ppbwz\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.281578 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.281558 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c13625d5-ppbwz\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.291474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.291451 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h"] Oct 30 18:47:22.291671 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:22.291652 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725f1e3c_099a_49fa_9cd7_f8e8bd3a910f.slice/crio-3511f5383ca6027f7d66e6a4f9fa1b9e5948e61526751fbd29f3b025b4d690b8 WatchSource:0}: Error finding container 3511f5383ca6027f7d66e6a4f9fa1b9e5948e61526751fbd29f3b025b4d690b8: Status 404 returned error can't find the container with id 3511f5383ca6027f7d66e6a4f9fa1b9e5948e61526751fbd29f3b025b4d690b8 Oct 30 18:47:22.298368 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.298350 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56gp\" (UniqueName: \"kubernetes.io/projected/26b8a80b-3296-47ff-9301-3a9ad215c6d2-kube-api-access-g56gp\") pod \"maas-api-revoke-sub-c13625d5-ppbwz\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.420620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.420590 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:22.626296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.624847 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" event={"ID":"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f","Type":"ContainerStarted","Data":"c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6"} Oct 30 18:47:22.626296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.624948 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" event={"ID":"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f","Type":"ContainerStarted","Data":"3511f5383ca6027f7d66e6a4f9fa1b9e5948e61526751fbd29f3b025b4d690b8"} Oct 30 18:47:22.626296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.625192 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:22.666857 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.666814 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" podStartSLOduration=1.6668008429999999 podStartE2EDuration="1.666800843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:22.662473245 +0000 UTC m=+2941.711518154" watchObservedRunningTime="2026-10-30 18:47:22.666800843 +0000 UTC m=+2941.715845783" Oct 30 18:47:22.743618 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:22.743594 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz"] Oct 30 18:47:22.743976 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:22.743795 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b8a80b_3296_47ff_9301_3a9ad215c6d2.slice/crio-3eb2de739a196fbebaa2fdf6486df1f5faa5a6abbd9e0e138e0621860a83cb0e WatchSource:0}: Error finding container 3eb2de739a196fbebaa2fdf6486df1f5faa5a6abbd9e0e138e0621860a83cb0e: Status 404 returned error can't find the container with id 3eb2de739a196fbebaa2fdf6486df1f5faa5a6abbd9e0e138e0621860a83cb0e Oct 30 18:47:23.632821 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:23.632789 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" event={"ID":"26b8a80b-3296-47ff-9301-3a9ad215c6d2","Type":"ContainerStarted","Data":"6b7594d8f5d02cbfca0d1507985c359927e3196c9faba469b34bbccfc623f214"} Oct 30 18:47:23.633005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:23.632841 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" event={"ID":"26b8a80b-3296-47ff-9301-3a9ad215c6d2","Type":"ContainerStarted","Data":"3eb2de739a196fbebaa2fdf6486df1f5faa5a6abbd9e0e138e0621860a83cb0e"} Oct 30 18:47:23.675943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:23.675890 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" podStartSLOduration=1.675877888 podStartE2EDuration="1.675877888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:23.673576705 +0000 UTC m=+2942.722621618" watchObservedRunningTime="2026-10-30 18:47:23.675877888 +0000 UTC m=+2942.724922801" Oct 30 18:47:23.805511 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:23.805471 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" podUID="c0f99068-6db8-431f-8aa3-578439c536ba" containerName="payload-processing" containerID="cri-o://bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03" gracePeriod=25 Oct 30 18:47:24.069679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.069654 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:47:24.203069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.202986 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-routing-overlay-volume\") pod \"c0f99068-6db8-431f-8aa3-578439c536ba\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " Oct 30 18:47:24.203069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.203058 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-plugins-config-volume\") pod \"c0f99068-6db8-431f-8aa3-578439c536ba\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " Oct 30 18:47:24.203341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.203093 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-kube-api-access-f789m\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-kube-api-access-f789m\") pod \"c0f99068-6db8-431f-8aa3-578439c536ba\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " Oct 30 18:47:24.203341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.203217 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-provider-credentials\") pod \"c0f99068-6db8-431f-8aa3-578439c536ba\" (UID: \"c0f99068-6db8-431f-8aa3-578439c536ba\") " Oct 30 18:47:24.203451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.203371 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-routing-overlay-volume" pod "c0f99068-6db8-431f-8aa3-578439c536ba" (UID: "c0f99068-6db8-431f-8aa3-578439c536ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:24.203500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.203461 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-plugins-config-volume" pod "c0f99068-6db8-431f-8aa3-578439c536ba" (UID: "c0f99068-6db8-431f-8aa3-578439c536ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:24.203762 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.203743 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:24.203762 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.203760 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0f99068-6db8-431f-8aa3-578439c536ba-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:24.205512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.205488 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-kube-api-access-f789m" pod "c0f99068-6db8-431f-8aa3-578439c536ba" (UID: "c0f99068-6db8-431f-8aa3-578439c536ba"). InnerVolumeSpecName "kube-api-access-f789m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:24.205601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.205566 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-provider-credentials" pod "c0f99068-6db8-431f-8aa3-578439c536ba" (UID: "c0f99068-6db8-431f-8aa3-578439c536ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:24.305026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.304998 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:24.305026 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.305023 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f789m\" (UniqueName: \"kubernetes.io/projected/c0f99068-6db8-431f-8aa3-578439c536ba-kube-api-access-f789m\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:24.641966 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.641922 4862 generic.go:356] "Generic (PLEG): container finished" podID="c0f99068-6db8-431f-8aa3-578439c536ba" containerID="bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03" exitCode=0 Oct 30 18:47:24.642132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.641987 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" event={"ID":"c0f99068-6db8-431f-8aa3-578439c536ba","Type":"ContainerDied","Data":"bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03"} Oct 30 18:47:24.642132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.641991 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" Oct 30 18:47:24.642132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.642048 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb" event={"ID":"c0f99068-6db8-431f-8aa3-578439c536ba","Type":"ContainerDied","Data":"3e53ffd67806022cf030010ce776f174d800ddce5a4121c668a87e63b02784db"} Oct 30 18:47:24.642132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.642079 4862 scope.go:122] "RemoveContainer" containerID="bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03" Oct 30 18:47:24.644101 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.644078 4862 generic.go:356] "Generic (PLEG): container finished" podID="26b8a80b-3296-47ff-9301-3a9ad215c6d2" containerID="6b7594d8f5d02cbfca0d1507985c359927e3196c9faba469b34bbccfc623f214" exitCode=0 Oct 30 18:47:24.644229 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.644107 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" event={"ID":"26b8a80b-3296-47ff-9301-3a9ad215c6d2","Type":"ContainerDied","Data":"6b7594d8f5d02cbfca0d1507985c359927e3196c9faba469b34bbccfc623f214"} Oct 30 18:47:24.653133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.653109 4862 scope.go:122] "RemoveContainer" containerID="bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03" Oct 30 18:47:24.653413 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:24.653392 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03\": container with ID starting with bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03 not found: ID does not exist" containerID="bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03" Oct 30 18:47:24.653492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.653424 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03"} err="failed to get container status \"bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03\": rpc error: code = NotFound desc = could not find container \"bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03\": container with ID starting with bdc1ffb37db62d6901975434e0ac47c29ccf007bb8f7b83a33bdcde2b08d3b03 not found: ID does not exist" Oct 30 18:47:24.734106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.734079 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb"] Oct 30 18:47:24.737667 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:24.737641 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-jkhqb"] Oct 30 18:47:25.063111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.063081 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f99068-6db8-431f-8aa3-578439c536ba" path="/var/lib/kubelet/pods/c0f99068-6db8-431f-8aa3-578439c536ba/volumes" Oct 30 18:47:25.539397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.539323 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l"] Oct 30 18:47:25.539948 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.539932 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c0f99068-6db8-431f-8aa3-578439c536ba" containerName="payload-processing" Oct 30 18:47:25.539988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.539950 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f99068-6db8-431f-8aa3-578439c536ba" containerName="payload-processing" Oct 30 18:47:25.540088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.540078 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f99068-6db8-431f-8aa3-578439c536ba" containerName="payload-processing" Oct 30 18:47:25.544922 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.544904 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.578546 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.578520 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l"] Oct 30 18:47:25.619224 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.619192 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.619411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.619294 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.619411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.619354 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.619544 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.619519 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6318be39-5955-4db1-99f2-318432ad0e0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.619605 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.619583 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.619871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.619852 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.619948 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.619905 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6318be39-5955-4db1-99f2-318432ad0e0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.620043 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.620028 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6318be39-5955-4db1-99f2-318432ad0e0d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.620080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.620060 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwmb\" (UniqueName: \"kubernetes.io/projected/6318be39-5955-4db1-99f2-318432ad0e0d-kube-api-access-9nwmb\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.721134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.721087 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6318be39-5955-4db1-99f2-318432ad0e0d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.721286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.721141 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwmb\" (UniqueName: \"kubernetes.io/projected/6318be39-5955-4db1-99f2-318432ad0e0d-kube-api-access-9nwmb\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.721286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.721179 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.721286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.721240 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.721471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.721318 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.721471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.721434 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6318be39-5955-4db1-99f2-318432ad0e0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.721582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.721487 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.721582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.721560 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.721682 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.721610 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6318be39-5955-4db1-99f2-318432ad0e0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.722484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.722401 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.723048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.722990 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.723402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.723379 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.723944 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.723895 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6318be39-5955-4db1-99f2-318432ad0e0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.724334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.724312 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.726109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.726082 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6318be39-5955-4db1-99f2-318432ad0e0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.728006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.727981 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6318be39-5955-4db1-99f2-318432ad0e0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.744099 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.744053 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6318be39-5955-4db1-99f2-318432ad0e0d-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.746000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.745976 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwmb\" (UniqueName: \"kubernetes.io/projected/6318be39-5955-4db1-99f2-318432ad0e0d-kube-api-access-9nwmb\") pod \"maas-default-gateway-openshift-default-76597d6855-46d6l\" (UID: \"6318be39-5955-4db1-99f2-318432ad0e0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.811824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.811801 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:25.859708 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.859680 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:25.923833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.923799 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-xks-ca\") pod \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " Oct 30 18:47:25.923993 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.923877 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b8a80b-3296-47ff-9301-3a9ad215c6d2-kube-api-access-g56gp\" (UniqueName: \"kubernetes.io/projected/26b8a80b-3296-47ff-9301-3a9ad215c6d2-kube-api-access-g56gp\") pod \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " Oct 30 18:47:25.924060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.924043 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-openshift-ca\") pod \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\" (UID: \"26b8a80b-3296-47ff-9301-3a9ad215c6d2\") " Oct 30 18:47:25.924931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.924905 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-openshift-ca" pod "26b8a80b-3296-47ff-9301-3a9ad215c6d2" (UID: "26b8a80b-3296-47ff-9301-3a9ad215c6d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.926923 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.926901 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-xks-ca" pod "26b8a80b-3296-47ff-9301-3a9ad215c6d2" (UID: "26b8a80b-3296-47ff-9301-3a9ad215c6d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.930230 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:25.930206 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b8a80b-3296-47ff-9301-3a9ad215c6d2-kube-api-access-g56gp" pod "26b8a80b-3296-47ff-9301-3a9ad215c6d2" (UID: "26b8a80b-3296-47ff-9301-3a9ad215c6d2"). InnerVolumeSpecName "kube-api-access-g56gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:26.026143 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.026116 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.026143 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.026142 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56gp\" (UniqueName: \"kubernetes.io/projected/26b8a80b-3296-47ff-9301-3a9ad215c6d2-kube-api-access-g56gp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.026143 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.026151 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b8a80b-3296-47ff-9301-3a9ad215c6d2-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.189687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.189658 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l"] Oct 30 18:47:26.190132 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:26.190111 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6318be39_5955_4db1_99f2_318432ad0e0d.slice/crio-3ac058d949519777d92934709a65a3fabdbc63210abca8b55289bf891ca6c046 WatchSource:0}: Error finding container 3ac058d949519777d92934709a65a3fabdbc63210abca8b55289bf891ca6c046: Status 404 returned error can't find the container with id 3ac058d949519777d92934709a65a3fabdbc63210abca8b55289bf891ca6c046 Oct 30 18:47:26.193318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.193280 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:26.193417 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.193361 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:26.193417 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.193404 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:26.660479 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.660450 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" Oct 30 18:47:26.660671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.660442 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz" event={"ID":"26b8a80b-3296-47ff-9301-3a9ad215c6d2","Type":"ContainerDied","Data":"3eb2de739a196fbebaa2fdf6486df1f5faa5a6abbd9e0e138e0621860a83cb0e"} Oct 30 18:47:26.660671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.660590 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb2de739a196fbebaa2fdf6486df1f5faa5a6abbd9e0e138e0621860a83cb0e" Oct 30 18:47:26.662223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.662196 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" event={"ID":"6318be39-5955-4db1-99f2-318432ad0e0d","Type":"ContainerStarted","Data":"0336159f0a59e28c69358234bb16079fd10f8045a040d6330f289b766d1738fc"} Oct 30 18:47:26.662357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.662234 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" event={"ID":"6318be39-5955-4db1-99f2-318432ad0e0d","Type":"ContainerStarted","Data":"3ac058d949519777d92934709a65a3fabdbc63210abca8b55289bf891ca6c046"} Oct 30 18:47:26.710337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.710293 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podStartSLOduration=1.710281897 podStartE2EDuration="1.710281897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:26.706116549 +0000 UTC m=+2945.755161462" watchObservedRunningTime="2026-10-30 18:47:26.710281897 +0000 UTC m=+2945.759326806" Oct 30 18:47:26.860524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:26.860492 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:27.873003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:27.872397 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-46d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:47:27.873003 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:27.872462 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podUID="6318be39-5955-4db1-99f2-318432ad0e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:47:28.867748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.867704 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns"] Oct 30 18:47:28.868678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.868320 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-46d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:47:28.868678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.868387 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podUID="6318be39-5955-4db1-99f2-318432ad0e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:47:28.868678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.868622 4862 cpu_manager.go:414] "containerMap: removing container" podUID="26b8a80b-3296-47ff-9301-3a9ad215c6d2" containerName="revoke-keys" Oct 30 18:47:28.868678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.868645 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b8a80b-3296-47ff-9301-3a9ad215c6d2" containerName="revoke-keys" Oct 30 18:47:28.868908 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.868793 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b8a80b-3296-47ff-9301-3a9ad215c6d2" containerName="revoke-keys" Oct 30 18:47:28.872907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.872886 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:28.874330 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.874305 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns"] Oct 30 18:47:28.887576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.887529 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"agc-cred-87de809eb6d3\"" Oct 30 18:47:28.887576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.887540 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-38e0db\"" Oct 30 18:47:28.887576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.887547 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"kube-root-ca.crt\"" Oct 30 18:47:28.887576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.887569 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"routing-overlay\"" Oct 30 18:47:28.897902 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.897884 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"openshift-service-ca.crt\"" Oct 30 18:47:28.899864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.899846 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"payload-processing-external-model-e2e--d1be02e2-dockercfg-jb7dz\"" Oct 30 18:47:28.989325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.969733 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-kube-api-access-p54qj\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:28.989325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.969932 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:28.989325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.969959 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:28.989325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:28.970009 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.074325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.071848 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-kube-api-access-p54qj\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.074325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.071996 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.074325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.072023 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.074325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.072064 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.074325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.073656 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.075041 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.074761 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.088209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.088183 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.092967 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.092942 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-kube-api-access-p54qj\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.189598 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.189451 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.670705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.670510 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns"] Oct 30 18:47:29.715216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.715192 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9"] Oct 30 18:47:29.720960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.720934 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.748717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.748691 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"payload-processing-external-model-e2e--41b9a229-dockercfg-dk66v\"" Oct 30 18:47:29.748943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.748927 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"agc-cred-62d3befb0e38\"" Oct 30 18:47:29.749166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.749149 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"routing-overlay\"" Oct 30 18:47:29.749366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.749350 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"openshift-service-ca.crt\"" Oct 30 18:47:29.749562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.749546 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2107d4\"" Oct 30 18:47:29.751475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.751445 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"kube-root-ca.crt\"" Oct 30 18:47:29.763303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.763280 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9"] Oct 30 18:47:29.793451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.792718 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.793451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.792834 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68j5v\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-kube-api-access-68j5v\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.793451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.792983 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.793451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.793187 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.871139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.871048 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-46d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:47:29.871439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.871390 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podUID="6318be39-5955-4db1-99f2-318432ad0e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:47:29.915353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.906600 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" event={"ID":"c684c0e8-e10e-434c-a7d8-cd0d38822e73","Type":"ContainerStarted","Data":"592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a"} Oct 30 18:47:29.915353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.906667 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" event={"ID":"c684c0e8-e10e-434c-a7d8-cd0d38822e73","Type":"ContainerStarted","Data":"5dbd517725a1d6e167c17247a42de9865b7bdb3dea17bea1aaebb79b7975f3d5"} Oct 30 18:47:29.915353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.907710 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:29.915353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.915176 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.916145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.916120 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.923458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.923419 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.941515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.924372 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.941750 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.937921 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h"] Oct 30 18:47:29.942136 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.916229 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.981600 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.941414 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68j5v\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-kube-api-access-68j5v\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.982906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.982831 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" podStartSLOduration=1.982811376 podStartE2EDuration="1.982811376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:29.979682666 +0000 UTC m=+2949.028727580" watchObservedRunningTime="2026-10-30 18:47:29.982811376 +0000 UTC m=+2949.031856290" Oct 30 18:47:29.987562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.987532 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:29.988164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:29.988137 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:30.001158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:30.001134 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68j5v\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-kube-api-access-68j5v\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:30.072000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:30.071972 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:30.519883 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:30.519852 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8060716_556f_4b67_9d34_0b3a6a3675ca.slice/crio-b39846b604fa12fa765cd20eab8638c5aba6fd9aa0f4e94a67b2709a32f779c0 WatchSource:0}: Error finding container b39846b604fa12fa765cd20eab8638c5aba6fd9aa0f4e94a67b2709a32f779c0: Status 404 returned error can't find the container with id b39846b604fa12fa765cd20eab8638c5aba6fd9aa0f4e94a67b2709a32f779c0 Oct 30 18:47:30.520217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:30.520192 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9"] Oct 30 18:47:30.860314 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:30.860285 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-46d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:47:30.860467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:30.860339 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podUID="6318be39-5955-4db1-99f2-318432ad0e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:47:30.915154 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:30.915125 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" event={"ID":"a8060716-556f-4b67-9d34-0b3a6a3675ca","Type":"ContainerStarted","Data":"e5d03105e88ca73a5b813401e1f0ffe20ca1a24947f5f3ea3a90193eeddd0ecf"} Oct 30 18:47:30.915341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:30.915176 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" event={"ID":"a8060716-556f-4b67-9d34-0b3a6a3675ca","Type":"ContainerStarted","Data":"b39846b604fa12fa765cd20eab8638c5aba6fd9aa0f4e94a67b2709a32f779c0"} Oct 30 18:47:30.915341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:30.915204 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:30.981649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:30.981600 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" podStartSLOduration=1.9815859599999999 podStartE2EDuration="1.98158596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:30.977810057 +0000 UTC m=+2950.026854969" watchObservedRunningTime="2026-10-30 18:47:30.98158596 +0000 UTC m=+2950.030630873" Oct 30 18:47:31.860128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:31.860093 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-46d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:47:31.860422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:31.860154 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podUID="6318be39-5955-4db1-99f2-318432ad0e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:47:32.860735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:32.860706 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-46d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:47:32.861153 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:32.860753 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podUID="6318be39-5955-4db1-99f2-318432ad0e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:47:33.860547 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:33.860516 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-46d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:47:33.860716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:33.860569 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podUID="6318be39-5955-4db1-99f2-318432ad0e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:47:34.942501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:34.942453 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" podUID="725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" containerName="payload-processing" containerID="cri-o://c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6" gracePeriod=25 Oct 30 18:47:35.249664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.249632 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:35.305625 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.305566 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-routing-overlay-volume\") pod \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " Oct 30 18:47:35.305625 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.305618 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-provider-credentials\") pod \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " Oct 30 18:47:35.305885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.305720 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-plugins-config-volume\") pod \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " Oct 30 18:47:35.305951 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.305903 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-kube-api-access-rrj2c\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-kube-api-access-rrj2c\") pod \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\" (UID: \"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f\") " Oct 30 18:47:35.306497 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.306463 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-routing-overlay-volume" pod "725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" (UID: "725f1e3c-099a-49fa-9cd7-f8e8bd3a910f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:35.307065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.307040 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.307065 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.307037 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-plugins-config-volume" pod "725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" (UID: "725f1e3c-099a-49fa-9cd7-f8e8bd3a910f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:35.311250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.311218 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-kube-api-access-rrj2c" pod "725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" (UID: "725f1e3c-099a-49fa-9cd7-f8e8bd3a910f"). InnerVolumeSpecName "kube-api-access-rrj2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:35.311616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.311567 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-provider-credentials" pod "725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" (UID: "725f1e3c-099a-49fa-9cd7-f8e8bd3a910f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:35.415867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.415833 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrj2c\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-kube-api-access-rrj2c\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.416143 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.416126 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.416295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.416280 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.663292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.663245 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2"] Oct 30 18:47:35.663865 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.663851 4862 cpu_manager.go:414] "containerMap: removing container" podUID="725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" containerName="payload-processing" Oct 30 18:47:35.663916 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.663867 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" containerName="payload-processing" Oct 30 18:47:35.663998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.663989 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" containerName="payload-processing" Oct 30 18:47:35.667597 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.667583 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.688353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.688326 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2"] Oct 30 18:47:35.720742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.720706 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.720935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.720785 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg2zp\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-kube-api-access-rg2zp\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.721028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.720950 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.721158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.721136 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.822051 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.822012 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.822251 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.822061 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg2zp\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-kube-api-access-rg2zp\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.822251 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.822117 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.822251 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.822211 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.822833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.822802 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.822966 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.822947 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.824713 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.824692 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.841151 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.841123 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg2zp\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-kube-api-access-rg2zp\") pod \"payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:35.917679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.911383 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-46d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:47:35.917679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.911451 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podUID="6318be39-5955-4db1-99f2-318432ad0e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:35.997288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:35.993907 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:36.016386 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.015799 4862 generic.go:356] "Generic (PLEG): container finished" podID="725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" containerID="c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6" exitCode=0 Oct 30 18:47:36.016386 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.016040 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" event={"ID":"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f","Type":"ContainerDied","Data":"c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6"} Oct 30 18:47:36.016386 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.016098 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" event={"ID":"725f1e3c-099a-49fa-9cd7-f8e8bd3a910f","Type":"ContainerDied","Data":"3511f5383ca6027f7d66e6a4f9fa1b9e5948e61526751fbd29f3b025b4d690b8"} Oct 30 18:47:36.016386 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.016130 4862 scope.go:122] "RemoveContainer" containerID="c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6" Oct 30 18:47:36.019147 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.016856 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h" Oct 30 18:47:36.073188 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.072850 4862 scope.go:122] "RemoveContainer" containerID="c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6" Oct 30 18:47:36.073752 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:36.073614 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6\": container with ID starting with c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6 not found: ID does not exist" containerID="c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6" Oct 30 18:47:36.073752 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.073656 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6"} err="failed to get container status \"c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6\": rpc error: code = NotFound desc = could not find container \"c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6\": container with ID starting with c1a59978ce3d6faad839a80c76b6b843b7807084f9494b944b449c21292464a6 not found: ID does not exist" Oct 30 18:47:36.116899 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.116864 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h"] Oct 30 18:47:36.147363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.144109 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d48d4766c-p8l8h"] Oct 30 18:47:36.337042 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.337009 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2"] Oct 30 18:47:36.338486 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:36.338452 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58604a08_f5b0_4fb8_8b45_817ce1219517.slice/crio-ceaaf82aa004d18b9988d8ae5b51a3e2802d10e71269aa5e5424893ce264419f WatchSource:0}: Error finding container ceaaf82aa004d18b9988d8ae5b51a3e2802d10e71269aa5e5424893ce264419f: Status 404 returned error can't find the container with id ceaaf82aa004d18b9988d8ae5b51a3e2802d10e71269aa5e5424893ce264419f Oct 30 18:47:36.953432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.953388 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-46d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:47:36.953641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:36.953438 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" podUID="6318be39-5955-4db1-99f2-318432ad0e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:37.024588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:37.024557 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" event={"ID":"58604a08-f5b0-4fb8-8b45-817ce1219517","Type":"ContainerStarted","Data":"6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4"} Oct 30 18:47:37.025024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:37.024607 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" event={"ID":"58604a08-f5b0-4fb8-8b45-817ce1219517","Type":"ContainerStarted","Data":"ceaaf82aa004d18b9988d8ae5b51a3e2802d10e71269aa5e5424893ce264419f"} Oct 30 18:47:37.025024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:37.024668 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:37.075158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:37.075117 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="725f1e3c-099a-49fa-9cd7-f8e8bd3a910f" path="/var/lib/kubelet/pods/725f1e3c-099a-49fa-9cd7-f8e8bd3a910f/volumes" Oct 30 18:47:37.081973 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:37.081916 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" podStartSLOduration=2.081897001 podStartE2EDuration="2.081897001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:37.073029279 +0000 UTC m=+2956.122074197" watchObservedRunningTime="2026-10-30 18:47:37.081897001 +0000 UTC m=+2956.130941915" Oct 30 18:47:37.212935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:37.212839 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:37.213322 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:37.213297 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:37.214082 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:37.214062 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-46d6l" Oct 30 18:47:38.157083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:38.157049 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2"] Oct 30 18:47:38.173733 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:38.173704 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-713dc8/agc-cred-157fc3a04bb6: secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:38.173733 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:38.173732 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2: secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:38.173962 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:38.173792 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials podName:58604a08-f5b0-4fb8-8b45-817ce1219517 nodeName:}" failed. No retries permitted until 2026-10-30 18:47:38.673771181 +0000 UTC m=+2957.722816098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" (UID: "58604a08-f5b0-4fb8-8b45-817ce1219517") : secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:38.680391 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:38.680364 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-713dc8/agc-cred-157fc3a04bb6: secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:38.680391 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:38.680388 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2: secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:38.680605 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:38.680443 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials podName:58604a08-f5b0-4fb8-8b45-817ce1219517 nodeName:}" failed. No retries permitted until 2026-10-30 18:47:39.68042673 +0000 UTC m=+2958.729471625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" (UID: "58604a08-f5b0-4fb8-8b45-817ce1219517") : secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:39.615202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.615169 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272"] Oct 30 18:47:39.620599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.620581 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.637184 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.637156 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272"] Oct 30 18:47:39.645401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.645379 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:47:39.645539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.645436 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:47:39.692515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.692488 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvddr\" (UniqueName: \"kubernetes.io/projected/3c85ed67-3c59-4379-aba4-188f55d9bc92-kube-api-access-hvddr\") pod \"maas-api-revoke-sub-44d54f07-7z272\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.692661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.692559 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44d54f07-7z272\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.692661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.692634 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44d54f07-7z272\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.692779 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:39.692765 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-713dc8/agc-cred-157fc3a04bb6: secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:39.692824 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:39.692781 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2: secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:39.692859 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:39.692831 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials podName:58604a08-f5b0-4fb8-8b45-817ce1219517 nodeName:}" failed. No retries permitted until 2026-10-30 18:47:41.692814058 +0000 UTC m=+2960.741858951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" (UID: "58604a08-f5b0-4fb8-8b45-817ce1219517") : secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:39.794275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.794231 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvddr\" (UniqueName: \"kubernetes.io/projected/3c85ed67-3c59-4379-aba4-188f55d9bc92-kube-api-access-hvddr\") pod \"maas-api-revoke-sub-44d54f07-7z272\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.794421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.794332 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44d54f07-7z272\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.794421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.794365 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44d54f07-7z272\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.795202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.795181 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44d54f07-7z272\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.797634 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.797578 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44d54f07-7z272\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.813440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.813421 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvddr\" (UniqueName: \"kubernetes.io/projected/3c85ed67-3c59-4379-aba4-188f55d9bc92-kube-api-access-hvddr\") pod \"maas-api-revoke-sub-44d54f07-7z272\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:39.934878 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:39.934818 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:40.237876 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:40.237849 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c85ed67_3c59_4379_aba4_188f55d9bc92.slice/crio-8db1aaca1bc2a95a7188c4e9500cf902daa2261c8c990bf8c3b20b757808ebc1 WatchSource:0}: Error finding container 8db1aaca1bc2a95a7188c4e9500cf902daa2261c8c990bf8c3b20b757808ebc1: Status 404 returned error can't find the container with id 8db1aaca1bc2a95a7188c4e9500cf902daa2261c8c990bf8c3b20b757808ebc1 Oct 30 18:47:40.238009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:40.237929 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272"] Oct 30 18:47:41.062832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:41.062795 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" event={"ID":"3c85ed67-3c59-4379-aba4-188f55d9bc92","Type":"ContainerStarted","Data":"f4418affcce0430ff6c674b2145b6ebda572b89559ea644fa9eb7cc41e4a61ac"} Oct 30 18:47:41.063222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:41.062854 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" event={"ID":"3c85ed67-3c59-4379-aba4-188f55d9bc92","Type":"ContainerStarted","Data":"8db1aaca1bc2a95a7188c4e9500cf902daa2261c8c990bf8c3b20b757808ebc1"} Oct 30 18:47:41.098950 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:41.098907 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" podStartSLOduration=2.098895662 podStartE2EDuration="2.098895662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:41.097974143 +0000 UTC m=+2960.147019055" watchObservedRunningTime="2026-10-30 18:47:41.098895662 +0000 UTC m=+2960.147940575" Oct 30 18:47:41.719298 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:41.719240 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-713dc8/agc-cred-157fc3a04bb6: secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:41.719298 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:41.719278 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2: secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:41.719544 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:41.719334 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials podName:58604a08-f5b0-4fb8-8b45-817ce1219517 nodeName:}" failed. No retries permitted until 2026-10-30 18:47:45.719318839 +0000 UTC m=+2964.768363730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" (UID: "58604a08-f5b0-4fb8-8b45-817ce1219517") : secret "agc-cred-157fc3a04bb6" not found Oct 30 18:47:41.925567 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:41.925540 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:47:41.925739 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:41.925607 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:47:42.065491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:42.065459 4862 generic.go:356] "Generic (PLEG): container finished" podID="3c85ed67-3c59-4379-aba4-188f55d9bc92" containerID="f4418affcce0430ff6c674b2145b6ebda572b89559ea644fa9eb7cc41e4a61ac" exitCode=0 Oct 30 18:47:42.065934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:42.065524 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" event={"ID":"3c85ed67-3c59-4379-aba4-188f55d9bc92","Type":"ContainerDied","Data":"f4418affcce0430ff6c674b2145b6ebda572b89559ea644fa9eb7cc41e4a61ac"} Oct 30 18:47:43.236019 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.235997 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:43.342699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.342630 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-openshift-ca\") pod \"3c85ed67-3c59-4379-aba4-188f55d9bc92\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " Oct 30 18:47:43.342846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.342710 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c85ed67-3c59-4379-aba4-188f55d9bc92-kube-api-access-hvddr\" (UniqueName: \"kubernetes.io/projected/3c85ed67-3c59-4379-aba4-188f55d9bc92-kube-api-access-hvddr\") pod \"3c85ed67-3c59-4379-aba4-188f55d9bc92\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " Oct 30 18:47:43.342846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.342780 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-xks-ca\") pod \"3c85ed67-3c59-4379-aba4-188f55d9bc92\" (UID: \"3c85ed67-3c59-4379-aba4-188f55d9bc92\") " Oct 30 18:47:43.343001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.342980 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-openshift-ca" pod "3c85ed67-3c59-4379-aba4-188f55d9bc92" (UID: "3c85ed67-3c59-4379-aba4-188f55d9bc92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:43.343303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.343255 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:43.345058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.345035 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-xks-ca" pod "3c85ed67-3c59-4379-aba4-188f55d9bc92" (UID: "3c85ed67-3c59-4379-aba4-188f55d9bc92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:43.345058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.345045 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c85ed67-3c59-4379-aba4-188f55d9bc92-kube-api-access-hvddr" pod "3c85ed67-3c59-4379-aba4-188f55d9bc92" (UID: "3c85ed67-3c59-4379-aba4-188f55d9bc92"). InnerVolumeSpecName "kube-api-access-hvddr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:43.444848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.444825 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c85ed67-3c59-4379-aba4-188f55d9bc92-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:43.444962 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:43.444852 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvddr\" (UniqueName: \"kubernetes.io/projected/3c85ed67-3c59-4379-aba4-188f55d9bc92-kube-api-access-hvddr\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:44.040498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.040456 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" podUID="58604a08-f5b0-4fb8-8b45-817ce1219517" containerName="payload-processing" containerID="cri-o://6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4" gracePeriod=25 Oct 30 18:47:44.078978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.078950 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" event={"ID":"3c85ed67-3c59-4379-aba4-188f55d9bc92","Type":"ContainerDied","Data":"8db1aaca1bc2a95a7188c4e9500cf902daa2261c8c990bf8c3b20b757808ebc1"} Oct 30 18:47:44.079108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.078997 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8db1aaca1bc2a95a7188c4e9500cf902daa2261c8c990bf8c3b20b757808ebc1" Oct 30 18:47:44.079108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.078995 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272" Oct 30 18:47:44.309736 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.309716 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:44.457607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.457580 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-routing-overlay-volume\") pod \"58604a08-f5b0-4fb8-8b45-817ce1219517\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " Oct 30 18:47:44.457782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.457634 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials\") pod \"58604a08-f5b0-4fb8-8b45-817ce1219517\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " Oct 30 18:47:44.457782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.457691 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-kube-api-access-rg2zp\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-kube-api-access-rg2zp\") pod \"58604a08-f5b0-4fb8-8b45-817ce1219517\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " Oct 30 18:47:44.457879 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.457861 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-plugins-config-volume\") pod \"58604a08-f5b0-4fb8-8b45-817ce1219517\" (UID: \"58604a08-f5b0-4fb8-8b45-817ce1219517\") " Oct 30 18:47:44.457970 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.457938 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-routing-overlay-volume" pod "58604a08-f5b0-4fb8-8b45-817ce1219517" (UID: "58604a08-f5b0-4fb8-8b45-817ce1219517"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:44.458212 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.458195 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-plugins-config-volume" pod "58604a08-f5b0-4fb8-8b45-817ce1219517" (UID: "58604a08-f5b0-4fb8-8b45-817ce1219517"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:44.458541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.458523 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:44.458608 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.458543 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58604a08-f5b0-4fb8-8b45-817ce1219517-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:44.460727 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.460705 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials" pod "58604a08-f5b0-4fb8-8b45-817ce1219517" (UID: "58604a08-f5b0-4fb8-8b45-817ce1219517"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:44.460727 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.460717 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-kube-api-access-rg2zp" pod "58604a08-f5b0-4fb8-8b45-817ce1219517" (UID: "58604a08-f5b0-4fb8-8b45-817ce1219517"). InnerVolumeSpecName "kube-api-access-rg2zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:44.569352 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.569235 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:44.569578 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:44.569558 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg2zp\" (UniqueName: \"kubernetes.io/projected/58604a08-f5b0-4fb8-8b45-817ce1219517-kube-api-access-rg2zp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:45.086712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:45.086682 4862 generic.go:356] "Generic (PLEG): container finished" podID="58604a08-f5b0-4fb8-8b45-817ce1219517" containerID="6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4" exitCode=0 Oct 30 18:47:45.086850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:45.086720 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" event={"ID":"58604a08-f5b0-4fb8-8b45-817ce1219517","Type":"ContainerDied","Data":"6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4"} Oct 30 18:47:45.086850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:45.086739 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" Oct 30 18:47:45.086850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:45.086768 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2" event={"ID":"58604a08-f5b0-4fb8-8b45-817ce1219517","Type":"ContainerDied","Data":"ceaaf82aa004d18b9988d8ae5b51a3e2802d10e71269aa5e5424893ce264419f"} Oct 30 18:47:45.086850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:45.086791 4862 scope.go:122] "RemoveContainer" containerID="6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4" Oct 30 18:47:45.097290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:45.097250 4862 scope.go:122] "RemoveContainer" containerID="6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4" Oct 30 18:47:45.097556 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:45.097534 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4\": container with ID starting with 6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4 not found: ID does not exist" containerID="6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4" Oct 30 18:47:45.097608 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:45.097565 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4"} err="failed to get container status \"6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4\": rpc error: code = NotFound desc = could not find container \"6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4\": container with ID starting with 6db43c95e88b2b90620cd869dc0371a6fd22f120e27366e947fbdb485f720df4 not found: ID does not exist" Oct 30 18:47:45.139362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:45.139331 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2"] Oct 30 18:47:45.142901 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:45.142881 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-6fbbwlfp2"] Oct 30 18:47:47.062636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:47.062606 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58604a08-f5b0-4fb8-8b45-817ce1219517" path="/var/lib/kubelet/pods/58604a08-f5b0-4fb8-8b45-817ce1219517/volumes" Oct 30 18:47:48.466949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.466922 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl"] Oct 30 18:47:48.467607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.467590 4862 cpu_manager.go:414] "containerMap: removing container" podUID="58604a08-f5b0-4fb8-8b45-817ce1219517" containerName="payload-processing" Oct 30 18:47:48.467664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.467608 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58604a08-f5b0-4fb8-8b45-817ce1219517" containerName="payload-processing" Oct 30 18:47:48.467664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.467617 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3c85ed67-3c59-4379-aba4-188f55d9bc92" containerName="revoke-keys" Oct 30 18:47:48.467664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.467623 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c85ed67-3c59-4379-aba4-188f55d9bc92" containerName="revoke-keys" Oct 30 18:47:48.467763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.467739 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="58604a08-f5b0-4fb8-8b45-817ce1219517" containerName="payload-processing" Oct 30 18:47:48.467763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.467747 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c85ed67-3c59-4379-aba4-188f55d9bc92" containerName="revoke-keys" Oct 30 18:47:48.472679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.472662 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.489120 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.489097 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl"] Oct 30 18:47:48.505303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.498340 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"payload-processing-external-model-e2e--dacb1c2b-dockercfg-vhv72\"" Oct 30 18:47:48.505303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.498345 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"agc-cred-3357f1dd84d4\"" Oct 30 18:47:48.505303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.498436 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"routing-overlay\"" Oct 30 18:47:48.505303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.498451 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"openshift-service-ca.crt\"" Oct 30 18:47:48.505303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.498525 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-861bb3\"" Oct 30 18:47:48.505303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.500101 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"kube-root-ca.crt\"" Oct 30 18:47:48.536946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.536916 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.537078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.536952 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmgj9\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-kube-api-access-fmgj9\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.537151 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.537122 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.537223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.537171 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.645699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.641762 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.645699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.641849 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmgj9\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-kube-api-access-fmgj9\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.645699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.642081 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.645699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.642163 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.645699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.643400 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.645699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.644025 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.658724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.658349 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.670660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.670624 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmgj9\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-kube-api-access-fmgj9\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:48.787009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:48.786940 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:49.094120 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:49.094082 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl"] Oct 30 18:47:49.094319 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:49.094293 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36cd8743_fcc7_4bd4_a082_97ab9a2037c8.slice/crio-f386a3605d4107ebecf0b973ce090c2f73a8063008e6cd9bca777c72329f2ebf WatchSource:0}: Error finding container f386a3605d4107ebecf0b973ce090c2f73a8063008e6cd9bca777c72329f2ebf: Status 404 returned error can't find the container with id f386a3605d4107ebecf0b973ce090c2f73a8063008e6cd9bca777c72329f2ebf Oct 30 18:47:49.118647 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:49.118619 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" event={"ID":"36cd8743-fcc7-4bd4-a082-97ab9a2037c8","Type":"ContainerStarted","Data":"f386a3605d4107ebecf0b973ce090c2f73a8063008e6cd9bca777c72329f2ebf"} Oct 30 18:47:49.869367 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:49.869325 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:47:50.112935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.112906 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8"] Oct 30 18:47:50.124649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.124568 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.136135 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.136101 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" event={"ID":"36cd8743-fcc7-4bd4-a082-97ab9a2037c8","Type":"ContainerStarted","Data":"b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41"} Oct 30 18:47:50.136250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.136204 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:47:50.137690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.137668 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8"] Oct 30 18:47:50.144242 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.144220 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"agc-cred-8403349d486c\"" Oct 30 18:47:50.144242 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.144256 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-e2e--6d8a874f-dockercfg-j6h5f\"" Oct 30 18:47:50.144442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.144346 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"kube-root-ca.crt\"" Oct 30 18:47:50.144442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.144388 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6018b4\"" Oct 30 18:47:50.151539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.151521 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"openshift-service-ca.crt\"" Oct 30 18:47:50.151701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.151598 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"routing-overlay\"" Oct 30 18:47:50.165981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.165948 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27v5\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-kube-api-access-c27v5\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.165981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.165978 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.166139 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.166106 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.166210 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.166151 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.218287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.217238 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" podStartSLOduration=2.217218598 podStartE2EDuration="2.217218598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:50.212204352 +0000 UTC m=+2969.261249266" watchObservedRunningTime="2026-10-30 18:47:50.217218598 +0000 UTC m=+2969.266263513" Oct 30 18:47:50.270345 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.270297 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.270558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.270396 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.270558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.270513 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27v5\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-kube-api-access-c27v5\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.271427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.271397 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.271555 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.270539 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.272225 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.272203 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.276584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.276557 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.299894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.299870 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27v5\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-kube-api-access-c27v5\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.443370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.443290 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:50.528441 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.528416 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg"] Oct 30 18:47:50.752987 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:50.752950 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2817015_caf9_4756_9866_ec3b73836809.slice/crio-bbeda18a2193b39ba1e3fdca2423c4286611ea2fc699e5676b59063bac42672f WatchSource:0}: Error finding container bbeda18a2193b39ba1e3fdca2423c4286611ea2fc699e5676b59063bac42672f: Status 404 returned error can't find the container with id bbeda18a2193b39ba1e3fdca2423c4286611ea2fc699e5676b59063bac42672f Oct 30 18:47:50.753236 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:50.753214 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8"] Oct 30 18:47:51.144891 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:51.144851 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" event={"ID":"f2817015-caf9-4756-9866-ec3b73836809","Type":"ContainerStarted","Data":"05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3"} Oct 30 18:47:51.145330 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:51.144914 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" event={"ID":"f2817015-caf9-4756-9866-ec3b73836809","Type":"ContainerStarted","Data":"bbeda18a2193b39ba1e3fdca2423c4286611ea2fc699e5676b59063bac42672f"} Oct 30 18:47:51.145330 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:51.145207 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:47:51.187525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:51.187482 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" podStartSLOduration=1.187467847 podStartE2EDuration="1.187467847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:51.184372606 +0000 UTC m=+2970.233417548" watchObservedRunningTime="2026-10-30 18:47:51.187467847 +0000 UTC m=+2970.236512759" Oct 30 18:47:52.389155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.389128 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv"] Oct 30 18:47:52.393843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.393821 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.412590 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.412549 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv"] Oct 30 18:47:52.427048 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.427024 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-861bb3\"/\"agc-cred-ab842f5d10de\"" Oct 30 18:47:52.500927 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.500894 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.501110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.500943 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.501110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.501079 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pt9d\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-kube-api-access-2pt9d\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.501207 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.501123 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.604145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.604113 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.604389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.604190 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.604389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.604298 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pt9d\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-kube-api-access-2pt9d\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.604389 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.604342 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.605223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.605192 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.605484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.605457 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.609902 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.609873 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.624205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.624176 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pt9d\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-kube-api-access-2pt9d\") pod \"payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:52.713737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:52.713640 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:53.064018 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.063977 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv"] Oct 30 18:47:53.064141 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:53.063990 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fe30b3_06b6_4fd5_a63d_76d131985e73.slice/crio-8740a25740fb7e4f59074ecfe2eae7b13f5e3156f41a9ab897f66efa4b32772b WatchSource:0}: Error finding container 8740a25740fb7e4f59074ecfe2eae7b13f5e3156f41a9ab897f66efa4b32772b: Status 404 returned error can't find the container with id 8740a25740fb7e4f59074ecfe2eae7b13f5e3156f41a9ab897f66efa4b32772b Oct 30 18:47:53.160635 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.160599 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" event={"ID":"65fe30b3-06b6-4fd5-a63d-76d131985e73","Type":"ContainerStarted","Data":"2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8"} Oct 30 18:47:53.160773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.160655 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" event={"ID":"65fe30b3-06b6-4fd5-a63d-76d131985e73","Type":"ContainerStarted","Data":"8740a25740fb7e4f59074ecfe2eae7b13f5e3156f41a9ab897f66efa4b32772b"} Oct 30 18:47:53.160773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.160698 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:47:53.202306 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.202239 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" podStartSLOduration=1.20222722 podStartE2EDuration="1.20222722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:53.198643956 +0000 UTC m=+2972.247688871" watchObservedRunningTime="2026-10-30 18:47:53.20222722 +0000 UTC m=+2972.251272133" Oct 30 18:47:53.271647 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.271620 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9"] Oct 30 18:47:53.275860 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.275845 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.292520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.292495 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"agc-cred-45d23ae5f8fc\"" Oct 30 18:47:53.293705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.293684 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9"] Oct 30 18:47:53.319937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.319872 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.319937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.319903 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh75t\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-kube-api-access-fh75t\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.320158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.320112 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.320211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.320161 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.421537 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.421505 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.421952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.421545 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh75t\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-kube-api-access-fh75t\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.421952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.421712 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.421952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.421749 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.422472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.422453 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.422545 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.422524 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.424195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.424176 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.441573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.441550 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh75t\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-kube-api-access-fh75t\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.590155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.590086 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:53.895311 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:53.895275 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc727e427_533b_4fbf_933c_701ed44b9e9e.slice/crio-0fa583805369d5c27df469264634d66a884c569f740ad4e67d495a5b1bfe2ae3 WatchSource:0}: Error finding container 0fa583805369d5c27df469264634d66a884c569f740ad4e67d495a5b1bfe2ae3: Status 404 returned error can't find the container with id 0fa583805369d5c27df469264634d66a884c569f740ad4e67d495a5b1bfe2ae3 Oct 30 18:47:53.896055 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:53.896029 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9"] Oct 30 18:47:54.175873 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:54.175784 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" event={"ID":"c727e427-533b-4fbf-933c-701ed44b9e9e","Type":"ContainerStarted","Data":"e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8"} Oct 30 18:47:54.175873 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:54.175846 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" event={"ID":"c727e427-533b-4fbf-933c-701ed44b9e9e","Type":"ContainerStarted","Data":"0fa583805369d5c27df469264634d66a884c569f740ad4e67d495a5b1bfe2ae3"} Oct 30 18:47:54.176119 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:54.175892 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:47:54.217371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:54.217328 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" podStartSLOduration=1.217316265 podStartE2EDuration="1.217316265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:54.21634932 +0000 UTC m=+2973.265394232" watchObservedRunningTime="2026-10-30 18:47:54.217316265 +0000 UTC m=+2973.266361177" Oct 30 18:47:55.529661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.529620 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" podUID="c00b9600-af60-49ca-b66f-d8f70e66afdc" containerName="payload-processing" containerID="cri-o://439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598" gracePeriod=25 Oct 30 18:47:55.693202 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:47:55.693172 4862 logging.go:55] [core] [Channel #3122 SubChannel #3123]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.155:50052", ServerName: "10.133.0.155:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.155:50052: connect: connection refused" Oct 30 18:47:55.809364 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.809338 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:47:55.863029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.863003 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-routing-overlay-volume\") pod \"c00b9600-af60-49ca-b66f-d8f70e66afdc\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " Oct 30 18:47:55.863193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.863097 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-plugins-config-volume\") pod \"c00b9600-af60-49ca-b66f-d8f70e66afdc\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " Oct 30 18:47:55.863193 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.863136 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-provider-credentials\") pod \"c00b9600-af60-49ca-b66f-d8f70e66afdc\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " Oct 30 18:47:55.863351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.863213 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-kube-api-access-tw72p\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-kube-api-access-tw72p\") pod \"c00b9600-af60-49ca-b66f-d8f70e66afdc\" (UID: \"c00b9600-af60-49ca-b66f-d8f70e66afdc\") " Oct 30 18:47:55.863483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.863456 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-routing-overlay-volume" pod "c00b9600-af60-49ca-b66f-d8f70e66afdc" (UID: "c00b9600-af60-49ca-b66f-d8f70e66afdc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:55.863596 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.863565 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-plugins-config-volume" pod "c00b9600-af60-49ca-b66f-d8f70e66afdc" (UID: "c00b9600-af60-49ca-b66f-d8f70e66afdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:55.863876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.863850 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:55.863939 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.863883 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b9600-af60-49ca-b66f-d8f70e66afdc-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:55.865788 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.865761 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-kube-api-access-tw72p" pod "c00b9600-af60-49ca-b66f-d8f70e66afdc" (UID: "c00b9600-af60-49ca-b66f-d8f70e66afdc"). InnerVolumeSpecName "kube-api-access-tw72p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:55.865959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.865941 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-provider-credentials" pod "c00b9600-af60-49ca-b66f-d8f70e66afdc" (UID: "c00b9600-af60-49ca-b66f-d8f70e66afdc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:55.977128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.976174 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:55.977128 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:55.976214 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw72p\" (UniqueName: \"kubernetes.io/projected/c00b9600-af60-49ca-b66f-d8f70e66afdc-kube-api-access-tw72p\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:47:56.191336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.191308 4862 generic.go:356] "Generic (PLEG): container finished" podID="c00b9600-af60-49ca-b66f-d8f70e66afdc" containerID="439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598" exitCode=0 Oct 30 18:47:56.191542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.191361 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" Oct 30 18:47:56.191542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.191388 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" event={"ID":"c00b9600-af60-49ca-b66f-d8f70e66afdc","Type":"ContainerDied","Data":"439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598"} Oct 30 18:47:56.191542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.191451 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" event={"ID":"c00b9600-af60-49ca-b66f-d8f70e66afdc","Type":"ContainerDied","Data":"ba8f9ebd754ddce81d703b19c264c687afdc97b8c4cce0342805a5f3b2d0a93c"} Oct 30 18:47:56.191542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.191484 4862 scope.go:122] "RemoveContainer" containerID="439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598" Oct 30 18:47:56.203748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.203735 4862 scope.go:122] "RemoveContainer" containerID="439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598" Oct 30 18:47:56.203990 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:47:56.203972 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598\": container with ID starting with 439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598 not found: ID does not exist" containerID="439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598" Oct 30 18:47:56.204034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.203997 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598"} err="failed to get container status \"439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598\": rpc error: code = NotFound desc = could not find container \"439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598\": container with ID starting with 439365d6fba63dbe4f9b80a76ff8ba6206a17b3c5cab2f96b0ef9e9cd74ee598 not found: ID does not exist" Oct 30 18:47:56.244582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.244559 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg"] Oct 30 18:47:56.248150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.248132 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg"] Oct 30 18:47:56.693296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:56.693245 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-713dc8/payload-processing-external-model-e2e-extmod-a-713dc8-56f5k7whg" podUID="c00b9600-af60-49ca-b66f-d8f70e66afdc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.155:50052\" within 1s: context deadline exceeded" Oct 30 18:47:57.063021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:47:57.062994 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00b9600-af60-49ca-b66f-d8f70e66afdc" path="/var/lib/kubelet/pods/c00b9600-af60-49ca-b66f-d8f70e66afdc/volumes" Oct 30 18:48:01.148398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:01.148365 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:48:01.963072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:01.963035 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:48:02.155864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:02.155834 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:48:04.030638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:04.030604 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57"] Oct 30 18:48:04.042475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:04.042448 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744a0ac7-vpt57"] Oct 30 18:48:04.178399 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:04.178371 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:48:04.266218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:04.266186 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl"] Oct 30 18:48:05.069973 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:05.069930 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d57ae37-021a-452b-a7d9-ee659651a258" path="/var/lib/kubelet/pods/2d57ae37-021a-452b-a7d9-ee659651a258/volumes" Oct 30 18:48:06.072472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:06.072434 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7jqjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:48:06.073004 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:06.072509 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:06.200227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:06.200111 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:48:06.290313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:06.286012 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8"] Oct 30 18:48:09.266660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.266624 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" podUID="36cd8743-fcc7-4bd4-a082-97ab9a2037c8" containerName="payload-processing" containerID="cri-o://b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41" gracePeriod=25 Oct 30 18:48:09.562886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.562855 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:48:09.663809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.663775 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-plugins-config-volume\") pod \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " Oct 30 18:48:09.663983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.663819 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-provider-credentials\") pod \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " Oct 30 18:48:09.663983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.663864 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-routing-overlay-volume\") pod \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " Oct 30 18:48:09.663983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.663944 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-kube-api-access-fmgj9\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-kube-api-access-fmgj9\") pod \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\" (UID: \"36cd8743-fcc7-4bd4-a082-97ab9a2037c8\") " Oct 30 18:48:09.664214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.664191 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-plugins-config-volume" pod "36cd8743-fcc7-4bd4-a082-97ab9a2037c8" (UID: "36cd8743-fcc7-4bd4-a082-97ab9a2037c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:09.664335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.664311 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-routing-overlay-volume" pod "36cd8743-fcc7-4bd4-a082-97ab9a2037c8" (UID: "36cd8743-fcc7-4bd4-a082-97ab9a2037c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:09.664555 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.664539 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:48:09.664615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.664559 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:48:09.666284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.666245 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-provider-credentials" pod "36cd8743-fcc7-4bd4-a082-97ab9a2037c8" (UID: "36cd8743-fcc7-4bd4-a082-97ab9a2037c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:09.666338 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.666308 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-kube-api-access-fmgj9" pod "36cd8743-fcc7-4bd4-a082-97ab9a2037c8" (UID: "36cd8743-fcc7-4bd4-a082-97ab9a2037c8"). InnerVolumeSpecName "kube-api-access-fmgj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:09.766092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.766062 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmgj9\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-kube-api-access-fmgj9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:48:09.766092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:09.766089 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36cd8743-fcc7-4bd4-a082-97ab9a2037c8-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:48:10.336797 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.336760 4862 generic.go:356] "Generic (PLEG): container finished" podID="36cd8743-fcc7-4bd4-a082-97ab9a2037c8" containerID="b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41" exitCode=0 Oct 30 18:48:10.337315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.336793 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" event={"ID":"36cd8743-fcc7-4bd4-a082-97ab9a2037c8","Type":"ContainerDied","Data":"b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41"} Oct 30 18:48:10.337315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.336839 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" Oct 30 18:48:10.337315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.336849 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl" event={"ID":"36cd8743-fcc7-4bd4-a082-97ab9a2037c8","Type":"ContainerDied","Data":"f386a3605d4107ebecf0b973ce090c2f73a8063008e6cd9bca777c72329f2ebf"} Oct 30 18:48:10.337315 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.336871 4862 scope.go:122] "RemoveContainer" containerID="b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41" Oct 30 18:48:10.348958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.348942 4862 scope.go:122] "RemoveContainer" containerID="b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41" Oct 30 18:48:10.349226 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:48:10.349205 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41\": container with ID starting with b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41 not found: ID does not exist" containerID="b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41" Oct 30 18:48:10.349296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.349233 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41"} err="failed to get container status \"b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41\": rpc error: code = NotFound desc = could not find container \"b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41\": container with ID starting with b63dd7a15a65374c2919bb0937e37a0db72a45735378638fc93ae13cc45dcf41 not found: ID does not exist" Oct 30 18:48:10.393373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.393350 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl"] Oct 30 18:48:10.405449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.405423 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-6ffdnshfl"] Oct 30 18:48:10.675817 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.675745 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7"] Oct 30 18:48:10.676464 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.676447 4862 cpu_manager.go:414] "containerMap: removing container" podUID="36cd8743-fcc7-4bd4-a082-97ab9a2037c8" containerName="payload-processing" Oct 30 18:48:10.676524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.676466 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cd8743-fcc7-4bd4-a082-97ab9a2037c8" containerName="payload-processing" Oct 30 18:48:10.676524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.676484 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c00b9600-af60-49ca-b66f-d8f70e66afdc" containerName="payload-processing" Oct 30 18:48:10.676524 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.676489 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00b9600-af60-49ca-b66f-d8f70e66afdc" containerName="payload-processing" Oct 30 18:48:10.676634 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.676622 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cd8743-fcc7-4bd4-a082-97ab9a2037c8" containerName="payload-processing" Oct 30 18:48:10.676673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.676637 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00b9600-af60-49ca-b66f-d8f70e66afdc" containerName="payload-processing" Oct 30 18:48:10.681461 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.681440 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.707341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.707312 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7"] Oct 30 18:48:10.776601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.776573 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.776746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.776622 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.776746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.776703 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxtsb\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-kube-api-access-lxtsb\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.776830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.776753 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.776830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.776789 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.776895 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.776857 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.776986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.776973 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.777030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.777011 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.777081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.777064 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878055 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878224 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878088 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxtsb\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-kube-api-access-lxtsb\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878224 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878126 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878224 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878159 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878224 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878178 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878337 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878389 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878415 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878484 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878540 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878667 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.878756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.878745 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.879032 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.879010 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.879138 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.879097 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.880663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.880641 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.880915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.880894 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.898876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.898851 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.899364 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.899346 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxtsb\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-kube-api-access-lxtsb\") pod \"maas-default-gateway-openshift-default-76597d6855-wwft7\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:10.995888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:10.995833 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:11.069661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.069594 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cd8743-fcc7-4bd4-a082-97ab9a2037c8" path="/var/lib/kubelet/pods/36cd8743-fcc7-4bd4-a082-97ab9a2037c8/volumes" Oct 30 18:48:11.286807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.286644 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" podUID="f2817015-caf9-4756-9866-ec3b73836809" containerName="payload-processing" containerID="cri-o://05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3" gracePeriod=25 Oct 30 18:48:11.351662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.351634 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7"] Oct 30 18:48:11.362148 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:48:11.362126 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f697e0_e7d3_4347_b6bf_aa4e2541f7ea.slice/crio-ac03dcdd6ead22074562e9aa2d966f283c950c81db332452c408c64c1230fe40 WatchSource:0}: Error finding container ac03dcdd6ead22074562e9aa2d966f283c950c81db332452c408c64c1230fe40: Status 404 returned error can't find the container with id ac03dcdd6ead22074562e9aa2d966f283c950c81db332452c408c64c1230fe40 Oct 30 18:48:11.394940 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.394904 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:11.395039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.394985 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:11.395039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.395025 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:11.556052 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.556023 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:48:11.586204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.586176 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-routing-overlay-volume\") pod \"f2817015-caf9-4756-9866-ec3b73836809\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " Oct 30 18:48:11.586448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.586228 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-provider-credentials\") pod \"f2817015-caf9-4756-9866-ec3b73836809\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " Oct 30 18:48:11.586574 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.586473 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-plugins-config-volume\") pod \"f2817015-caf9-4756-9866-ec3b73836809\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " Oct 30 18:48:11.586651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.586630 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-kube-api-access-c27v5\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-kube-api-access-c27v5\") pod \"f2817015-caf9-4756-9866-ec3b73836809\" (UID: \"f2817015-caf9-4756-9866-ec3b73836809\") " Oct 30 18:48:11.586811 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.586789 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-routing-overlay-volume" pod "f2817015-caf9-4756-9866-ec3b73836809" (UID: "f2817015-caf9-4756-9866-ec3b73836809"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:11.586928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.586909 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-plugins-config-volume" pod "f2817015-caf9-4756-9866-ec3b73836809" (UID: "f2817015-caf9-4756-9866-ec3b73836809"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:11.587390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.587370 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.587508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.587394 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2817015-caf9-4756-9866-ec3b73836809-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.588892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.588872 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-provider-credentials" pod "f2817015-caf9-4756-9866-ec3b73836809" (UID: "f2817015-caf9-4756-9866-ec3b73836809"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:11.589012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.588997 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-kube-api-access-c27v5" pod "f2817015-caf9-4756-9866-ec3b73836809" (UID: "f2817015-caf9-4756-9866-ec3b73836809"). InnerVolumeSpecName "kube-api-access-c27v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:11.688420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.688386 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c27v5\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-kube-api-access-c27v5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.688420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:11.688414 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2817015-caf9-4756-9866-ec3b73836809-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:48:12.362817 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.362781 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" event={"ID":"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea","Type":"ContainerStarted","Data":"006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16"} Oct 30 18:48:12.363446 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.362846 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" event={"ID":"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea","Type":"ContainerStarted","Data":"ac03dcdd6ead22074562e9aa2d966f283c950c81db332452c408c64c1230fe40"} Oct 30 18:48:12.364388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.364366 4862 generic.go:356] "Generic (PLEG): container finished" podID="f2817015-caf9-4756-9866-ec3b73836809" containerID="05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3" exitCode=0 Oct 30 18:48:12.364470 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.364407 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" Oct 30 18:48:12.364470 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.364414 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" event={"ID":"f2817015-caf9-4756-9866-ec3b73836809","Type":"ContainerDied","Data":"05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3"} Oct 30 18:48:12.364470 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.364451 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8" event={"ID":"f2817015-caf9-4756-9866-ec3b73836809","Type":"ContainerDied","Data":"bbeda18a2193b39ba1e3fdca2423c4286611ea2fc699e5676b59063bac42672f"} Oct 30 18:48:12.364585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.364473 4862 scope.go:122] "RemoveContainer" containerID="05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3" Oct 30 18:48:12.375627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.375610 4862 scope.go:122] "RemoveContainer" containerID="05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3" Oct 30 18:48:12.375876 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:48:12.375853 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3\": container with ID starting with 05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3 not found: ID does not exist" containerID="05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3" Oct 30 18:48:12.375945 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.375886 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3"} err="failed to get container status \"05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3\": rpc error: code = NotFound desc = could not find container \"05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3\": container with ID starting with 05a708d2ab628e8588f375d01867249d7cbd8d7b14edd0f73cf9babc1acad2a3 not found: ID does not exist" Oct 30 18:48:12.419644 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.419604 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podStartSLOduration=2.419592382 podStartE2EDuration="2.419592382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:12.418010739 +0000 UTC m=+2991.467055652" watchObservedRunningTime="2026-10-30 18:48:12.419592382 +0000 UTC m=+2991.468637294" Oct 30 18:48:12.459836 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.459805 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8"] Oct 30 18:48:12.463992 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.463967 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-789fzsmj8"] Oct 30 18:48:12.996658 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.996620 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:12.998520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.998496 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wwft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:48:12.998630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:12.998548 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 18:48:13.078759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:13.078727 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2817015-caf9-4756-9866-ec3b73836809" path="/var/lib/kubelet/pods/f2817015-caf9-4756-9866-ec3b73836809/volumes" Oct 30 18:48:13.997122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:13.997036 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wwft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:48:13.997122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:13.997097 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 18:48:14.996801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:14.996770 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wwft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:48:14.996965 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:14.996822 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 18:48:15.996626 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:15.996594 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wwft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:48:15.997036 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:15.996644 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 18:48:16.996499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:16.996468 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wwft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:48:16.996673 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:16.996526 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 18:48:17.996832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:17.996802 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wwft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:48:17.997286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:17.996851 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 18:48:18.996961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:18.996923 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wwft7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 18:48:18.997406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:18.996976 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 18:48:19.875543 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:48:19.875504 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:48:20.001119 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:20.001068 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:20.001551 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:20.001535 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:20.002279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:20.002242 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:48:21.638717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:21.638690 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:48:21.639601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:21.639576 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:48:25.719662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.719631 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc"] Oct 30 18:48:25.720300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.720283 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f2817015-caf9-4756-9866-ec3b73836809" containerName="payload-processing" Oct 30 18:48:25.720300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.720300 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2817015-caf9-4756-9866-ec3b73836809" containerName="payload-processing" Oct 30 18:48:25.720435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.720426 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2817015-caf9-4756-9866-ec3b73836809" containerName="payload-processing" Oct 30 18:48:25.725471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.725455 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.777450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.777419 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc"] Oct 30 18:48:25.795958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.795921 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.796118 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.796035 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.796118 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.796113 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.796256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.796168 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.796256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.796201 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrzn\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-kube-api-access-9wrzn\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.796256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.796241 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88a089e-f502-464b-9a53-791d50e8c2f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.796451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.796347 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.796507 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.796448 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.796507 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.796482 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.897981 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898037 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898065 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrzn\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-kube-api-access-9wrzn\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898164 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898099 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88a089e-f502-464b-9a53-791d50e8c2f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898163 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898242 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898308 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898357 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898606 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898433 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898606 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898454 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898727 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898699 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898783 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898750 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.898843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898817 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.899071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.898897 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88a089e-f502-464b-9a53-791d50e8c2f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.900622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.900597 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.900728 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.900711 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.924701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.924603 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrzn\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-kube-api-access-9wrzn\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:25.930286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:25.930231 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-t7sdc\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:26.079174 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:26.079146 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:26.397179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:26.397150 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc"] Oct 30 18:48:26.397517 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:48:26.397495 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88a089e_f502_464b_9a53_791d50e8c2f2.slice/crio-1530329afbe3f05278f3421ed33cfcfebf39e47751de5b0704c99024f86095d1 WatchSource:0}: Error finding container 1530329afbe3f05278f3421ed33cfcfebf39e47751de5b0704c99024f86095d1: Status 404 returned error can't find the container with id 1530329afbe3f05278f3421ed33cfcfebf39e47751de5b0704c99024f86095d1 Oct 30 18:48:26.400232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:26.400201 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:26.400340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:26.400283 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:26.400340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:26.400329 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:26.475672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:26.475635 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" event={"ID":"d88a089e-f502-464b-9a53-791d50e8c2f2","Type":"ContainerStarted","Data":"5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8"} Oct 30 18:48:26.475792 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:26.475707 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" event={"ID":"d88a089e-f502-464b-9a53-791d50e8c2f2","Type":"ContainerStarted","Data":"1530329afbe3f05278f3421ed33cfcfebf39e47751de5b0704c99024f86095d1"} Oct 30 18:48:26.519662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:26.519616 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podStartSLOduration=1.519597247 podStartE2EDuration="1.519597247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:26.517235917 +0000 UTC m=+3005.566280830" watchObservedRunningTime="2026-10-30 18:48:26.519597247 +0000 UTC m=+3005.568642163" Oct 30 18:48:27.079343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:27.079311 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:27.081142 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:27.081116 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-t7sdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:48:27.081283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:27.081172 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 18:48:28.080380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:28.080349 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-t7sdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:48:28.080757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:28.080408 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 18:48:29.080473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:29.080435 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-t7sdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:48:29.080952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:29.080509 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 18:48:30.080199 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:30.080166 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-t7sdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:48:30.080388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:30.080220 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 18:48:31.079958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:31.079926 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-t7sdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:48:31.080400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:31.079980 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 18:48:32.079567 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:32.079530 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-t7sdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:48:32.079746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:32.079573 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 18:48:33.079496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:33.079467 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-t7sdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:48:33.079933 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:33.079513 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 18:48:34.079672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:34.079641 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-t7sdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 18:48:34.080093 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:34.079689 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 18:48:35.339988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:35.339955 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:35.341346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:35.341317 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:35.342204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:35.342184 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:48:38.988720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:38.988677 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j"] Oct 30 18:48:38.994824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:38.994786 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:39.011621 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.011596 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j"] Oct 30 18:48:39.018341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.017237 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0cc6b1-dockercfg-whmp2\"" Oct 30 18:48:39.018341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.017295 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0cc6b1\"" Oct 30 18:48:39.114911 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.114867 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2"] Oct 30 18:48:39.187430 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.187379 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2"] Oct 30 18:48:39.187765 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.187748 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:39.199964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.199920 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxqvp\" (UniqueName: \"kubernetes.io/projected/3e9466f2-fd67-4562-b757-8a43a6757a03-kube-api-access-fxqvp\") pod \"payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j\" (UID: \"3e9466f2-fd67-4562-b757-8a43a6757a03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:39.200357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.200327 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9466f2-fd67-4562-b757-8a43a6757a03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j\" (UID: \"3e9466f2-fd67-4562-b757-8a43a6757a03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:39.301821 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.301791 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxsk\" (UniqueName: \"kubernetes.io/projected/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-kube-api-access-8cxsk\") pod \"payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2\" (UID: \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:39.302006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.301900 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9466f2-fd67-4562-b757-8a43a6757a03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j\" (UID: \"3e9466f2-fd67-4562-b757-8a43a6757a03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:39.302085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.302004 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2\" (UID: \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:39.302219 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.302201 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxqvp\" (UniqueName: \"kubernetes.io/projected/3e9466f2-fd67-4562-b757-8a43a6757a03-kube-api-access-fxqvp\") pod \"payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j\" (UID: \"3e9466f2-fd67-4562-b757-8a43a6757a03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:39.302639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.302617 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9466f2-fd67-4562-b757-8a43a6757a03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j\" (UID: \"3e9466f2-fd67-4562-b757-8a43a6757a03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:39.322454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.322425 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxqvp\" (UniqueName: \"kubernetes.io/projected/3e9466f2-fd67-4562-b757-8a43a6757a03-kube-api-access-fxqvp\") pod \"payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j\" (UID: \"3e9466f2-fd67-4562-b757-8a43a6757a03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:39.404025 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.403985 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2\" (UID: \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:39.404221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.404208 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxsk\" (UniqueName: \"kubernetes.io/projected/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-kube-api-access-8cxsk\") pod \"payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2\" (UID: \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:39.404833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.404807 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2\" (UID: \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:39.424496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.424469 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxsk\" (UniqueName: \"kubernetes.io/projected/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-kube-api-access-8cxsk\") pod \"payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2\" (UID: \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:39.532680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.532650 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:39.616604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.616417 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:39.840818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.840793 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2"] Oct 30 18:48:39.841529 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:48:39.841475 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccf86cb_6638_4e77_a72c_5ba2a0f4e7b9.slice/crio-f9a0eb6d920128a26953d8634eba87751369305973ee15a00d57ffcbf044466a WatchSource:0}: Error finding container f9a0eb6d920128a26953d8634eba87751369305973ee15a00d57ffcbf044466a: Status 404 returned error can't find the container with id f9a0eb6d920128a26953d8634eba87751369305973ee15a00d57ffcbf044466a Oct 30 18:48:39.931017 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:48:39.930990 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9466f2_fd67_4562_b757_8a43a6757a03.slice/crio-edd0f1f2bff74f76090870f1be1591f370767a6ccc983afe9cf9e920b7124619 WatchSource:0}: Error finding container edd0f1f2bff74f76090870f1be1591f370767a6ccc983afe9cf9e920b7124619: Status 404 returned error can't find the container with id edd0f1f2bff74f76090870f1be1591f370767a6ccc983afe9cf9e920b7124619 Oct 30 18:48:39.931919 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:39.931889 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j"] Oct 30 18:48:40.724952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:40.724822 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" event={"ID":"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9","Type":"ContainerStarted","Data":"8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c"} Oct 30 18:48:40.724952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:40.724895 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:40.724952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:40.724920 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" event={"ID":"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9","Type":"ContainerStarted","Data":"f9a0eb6d920128a26953d8634eba87751369305973ee15a00d57ffcbf044466a"} Oct 30 18:48:40.727000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:40.726962 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" event={"ID":"3e9466f2-fd67-4562-b757-8a43a6757a03","Type":"ContainerStarted","Data":"00a4abacc5e63b8b5051c41b8969cd95fc278a5f82d4ad52e897913c4a656768"} Oct 30 18:48:40.727123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:40.727019 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" event={"ID":"3e9466f2-fd67-4562-b757-8a43a6757a03","Type":"ContainerStarted","Data":"edd0f1f2bff74f76090870f1be1591f370767a6ccc983afe9cf9e920b7124619"} Oct 30 18:48:40.727123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:40.727048 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:40.768103 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:40.768054 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" podStartSLOduration=1.7680387039999998 podStartE2EDuration="1.768038704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:40.765504178 +0000 UTC m=+3019.814549092" watchObservedRunningTime="2026-10-30 18:48:40.768038704 +0000 UTC m=+3019.817083611" Oct 30 18:48:40.803215 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:40.803171 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" podStartSLOduration=2.8031618910000002 podStartE2EDuration="2.803161891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:40.802513162 +0000 UTC m=+3019.851558076" watchObservedRunningTime="2026-10-30 18:48:40.803161891 +0000 UTC m=+3019.852206804" Oct 30 18:48:49.870742 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:48:49.870685 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:48:51.738206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:51.738176 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:48:51.738611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:51.738243 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:48:59.253092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:48:59.253060 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:49:03.941960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:03.941925 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn"] Oct 30 18:49:03.946517 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:03.946501 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:03.965123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:03.965087 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn"] Oct 30 18:49:03.972172 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:03.972135 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82edc819-af71-490d-8eec-289d197b4469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-056faee6-286mn\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:03.972324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:03.972199 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82edc819-af71-490d-8eec-289d197b4469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-056faee6-286mn\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:03.972404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:03.972328 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:03.972404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:03.972375 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:49:03.972576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:03.972556 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlvd\" (UniqueName: \"kubernetes.io/projected/82edc819-af71-490d-8eec-289d197b4469-kube-api-access-hhlvd\") pod \"maas-api-revoke-sub-056faee6-286mn\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:04.073896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.073858 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlvd\" (UniqueName: \"kubernetes.io/projected/82edc819-af71-490d-8eec-289d197b4469-kube-api-access-hhlvd\") pod \"maas-api-revoke-sub-056faee6-286mn\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:04.074097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.073975 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82edc819-af71-490d-8eec-289d197b4469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-056faee6-286mn\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:04.074097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.074022 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82edc819-af71-490d-8eec-289d197b4469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-056faee6-286mn\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:04.074627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.074608 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82edc819-af71-490d-8eec-289d197b4469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-056faee6-286mn\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:04.076627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.076604 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82edc819-af71-490d-8eec-289d197b4469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-056faee6-286mn\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:04.093107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.093074 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlvd\" (UniqueName: \"kubernetes.io/projected/82edc819-af71-490d-8eec-289d197b4469-kube-api-access-hhlvd\") pod \"maas-api-revoke-sub-056faee6-286mn\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:04.261010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.260942 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:04.564883 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.564848 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn"] Oct 30 18:49:04.568818 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:49:04.568790 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82edc819_af71_490d_8eec_289d197b4469.slice/crio-263907ac675bb548b5a5322de1cfbcdf645ca5a289ff04251e16824a7f6c548c WatchSource:0}: Error finding container 263907ac675bb548b5a5322de1cfbcdf645ca5a289ff04251e16824a7f6c548c: Status 404 returned error can't find the container with id 263907ac675bb548b5a5322de1cfbcdf645ca5a289ff04251e16824a7f6c548c Oct 30 18:49:04.915997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.915961 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" event={"ID":"82edc819-af71-490d-8eec-289d197b4469","Type":"ContainerStarted","Data":"96caa73dbb63cbdedfe8ea04707f9f8ab636cbac1527c303b6876d166766ef5a"} Oct 30 18:49:04.916170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.916029 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" event={"ID":"82edc819-af71-490d-8eec-289d197b4469","Type":"ContainerStarted","Data":"263907ac675bb548b5a5322de1cfbcdf645ca5a289ff04251e16824a7f6c548c"} Oct 30 18:49:04.961525 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:04.961475 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" podStartSLOduration=1.96146248 podStartE2EDuration="1.96146248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:04.960414149 +0000 UTC m=+3044.009459098" watchObservedRunningTime="2026-10-30 18:49:04.96146248 +0000 UTC m=+3044.010507417" Oct 30 18:49:05.924878 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:05.924847 4862 generic.go:356] "Generic (PLEG): container finished" podID="82edc819-af71-490d-8eec-289d197b4469" containerID="96caa73dbb63cbdedfe8ea04707f9f8ab636cbac1527c303b6876d166766ef5a" exitCode=0 Oct 30 18:49:05.925063 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:05.924919 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" event={"ID":"82edc819-af71-490d-8eec-289d197b4469","Type":"ContainerDied","Data":"96caa73dbb63cbdedfe8ea04707f9f8ab636cbac1527c303b6876d166766ef5a"} Oct 30 18:49:07.135998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.135976 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:07.214649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.214575 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82edc819-af71-490d-8eec-289d197b4469-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82edc819-af71-490d-8eec-289d197b4469-maas-api-openshift-ca\") pod \"82edc819-af71-490d-8eec-289d197b4469\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " Oct 30 18:49:07.214649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.214621 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82edc819-af71-490d-8eec-289d197b4469-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82edc819-af71-490d-8eec-289d197b4469-maas-api-xks-ca\") pod \"82edc819-af71-490d-8eec-289d197b4469\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " Oct 30 18:49:07.214868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.214752 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82edc819-af71-490d-8eec-289d197b4469-kube-api-access-hhlvd\" (UniqueName: \"kubernetes.io/projected/82edc819-af71-490d-8eec-289d197b4469-kube-api-access-hhlvd\") pod \"82edc819-af71-490d-8eec-289d197b4469\" (UID: \"82edc819-af71-490d-8eec-289d197b4469\") " Oct 30 18:49:07.214982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.214957 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82edc819-af71-490d-8eec-289d197b4469-maas-api-openshift-ca" pod "82edc819-af71-490d-8eec-289d197b4469" (UID: "82edc819-af71-490d-8eec-289d197b4469"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:07.215501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.215472 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82edc819-af71-490d-8eec-289d197b4469-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:07.217443 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.217416 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82edc819-af71-490d-8eec-289d197b4469-kube-api-access-hhlvd" pod "82edc819-af71-490d-8eec-289d197b4469" (UID: "82edc819-af71-490d-8eec-289d197b4469"). InnerVolumeSpecName "kube-api-access-hhlvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:07.217666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.217649 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82edc819-af71-490d-8eec-289d197b4469-maas-api-xks-ca" pod "82edc819-af71-490d-8eec-289d197b4469" (UID: "82edc819-af71-490d-8eec-289d197b4469"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:07.316380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.316354 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82edc819-af71-490d-8eec-289d197b4469-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:07.316380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.316377 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhlvd\" (UniqueName: \"kubernetes.io/projected/82edc819-af71-490d-8eec-289d197b4469-kube-api-access-hhlvd\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:07.941564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.941529 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" Oct 30 18:49:07.941564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.941552 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn" event={"ID":"82edc819-af71-490d-8eec-289d197b4469","Type":"ContainerDied","Data":"263907ac675bb548b5a5322de1cfbcdf645ca5a289ff04251e16824a7f6c548c"} Oct 30 18:49:07.941777 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:07.941626 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="263907ac675bb548b5a5322de1cfbcdf645ca5a289ff04251e16824a7f6c548c" Oct 30 18:49:14.723687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.723653 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd"] Oct 30 18:49:14.724330 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.724315 4862 cpu_manager.go:414] "containerMap: removing container" podUID="82edc819-af71-490d-8eec-289d197b4469" containerName="revoke-keys" Oct 30 18:49:14.724384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.724332 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82edc819-af71-490d-8eec-289d197b4469" containerName="revoke-keys" Oct 30 18:49:14.724448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.724440 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="82edc819-af71-490d-8eec-289d197b4469" containerName="revoke-keys" Oct 30 18:49:14.729355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.729338 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:14.747661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.747640 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd"] Oct 30 18:49:14.755080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.755057 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:14.755222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.755112 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:49:14.816582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.816557 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c8d63a5-72dqd\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:14.816693 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.816617 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4v7m\" (UniqueName: \"kubernetes.io/projected/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-kube-api-access-k4v7m\") pod \"maas-api-revoke-sub-3c8d63a5-72dqd\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:14.816870 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.816848 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c8d63a5-72dqd\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:14.918483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.918452 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c8d63a5-72dqd\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:14.918607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.918502 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c8d63a5-72dqd\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:14.918607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.918547 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4v7m\" (UniqueName: \"kubernetes.io/projected/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-kube-api-access-k4v7m\") pod \"maas-api-revoke-sub-3c8d63a5-72dqd\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:14.919210 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.919168 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c8d63a5-72dqd\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:14.921054 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.921036 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c8d63a5-72dqd\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:14.937995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:14.937967 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4v7m\" (UniqueName: \"kubernetes.io/projected/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-kube-api-access-k4v7m\") pod \"maas-api-revoke-sub-3c8d63a5-72dqd\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:15.044067 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:15.043997 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:15.349444 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:49:15.349416 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c6d33c_5374_4ad0_b00d_60c1ba64dec7.slice/crio-cb1421af7203825d332013220b52ba05196acda1a2c828233236e3b0e4713834 WatchSource:0}: Error finding container cb1421af7203825d332013220b52ba05196acda1a2c828233236e3b0e4713834: Status 404 returned error can't find the container with id cb1421af7203825d332013220b52ba05196acda1a2c828233236e3b0e4713834 Oct 30 18:49:15.349790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:15.349770 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd"] Oct 30 18:49:16.003010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:16.002928 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" event={"ID":"09c6d33c-5374-4ad0-b00d-60c1ba64dec7","Type":"ContainerStarted","Data":"ada9954f67e0a585a75107333af0b121ea37ac2f1896f65fb278d6384b0ba511"} Oct 30 18:49:16.003010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:16.002986 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" event={"ID":"09c6d33c-5374-4ad0-b00d-60c1ba64dec7","Type":"ContainerStarted","Data":"cb1421af7203825d332013220b52ba05196acda1a2c828233236e3b0e4713834"} Oct 30 18:49:16.045049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:16.045000 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" podStartSLOduration=2.044987352 podStartE2EDuration="2.044987352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:16.043654253 +0000 UTC m=+3055.092699165" watchObservedRunningTime="2026-10-30 18:49:16.044987352 +0000 UTC m=+3055.094032266" Oct 30 18:49:17.011773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:17.011733 4862 generic.go:356] "Generic (PLEG): container finished" podID="09c6d33c-5374-4ad0-b00d-60c1ba64dec7" containerID="ada9954f67e0a585a75107333af0b121ea37ac2f1896f65fb278d6384b0ba511" exitCode=0 Oct 30 18:49:17.012181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:17.011827 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" event={"ID":"09c6d33c-5374-4ad0-b00d-60c1ba64dec7","Type":"ContainerDied","Data":"ada9954f67e0a585a75107333af0b121ea37ac2f1896f65fb278d6384b0ba511"} Oct 30 18:49:18.205759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.205737 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:18.365141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.365059 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-xks-ca\") pod \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " Oct 30 18:49:18.365141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.365100 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-kube-api-access-k4v7m\" (UniqueName: \"kubernetes.io/projected/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-kube-api-access-k4v7m\") pod \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " Oct 30 18:49:18.365408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.365159 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-openshift-ca\") pod \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\" (UID: \"09c6d33c-5374-4ad0-b00d-60c1ba64dec7\") " Oct 30 18:49:18.365575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.365548 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-openshift-ca" pod "09c6d33c-5374-4ad0-b00d-60c1ba64dec7" (UID: "09c6d33c-5374-4ad0-b00d-60c1ba64dec7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:18.365844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.365825 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:18.367357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.367334 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-kube-api-access-k4v7m" pod "09c6d33c-5374-4ad0-b00d-60c1ba64dec7" (UID: "09c6d33c-5374-4ad0-b00d-60c1ba64dec7"). InnerVolumeSpecName "kube-api-access-k4v7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:18.367423 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.367406 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-xks-ca" pod "09c6d33c-5374-4ad0-b00d-60c1ba64dec7" (UID: "09c6d33c-5374-4ad0-b00d-60c1ba64dec7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:18.467381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.467348 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:18.467381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:18.467373 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4v7m\" (UniqueName: \"kubernetes.io/projected/09c6d33c-5374-4ad0-b00d-60c1ba64dec7-kube-api-access-k4v7m\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:19.055107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:19.055077 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" Oct 30 18:49:19.063303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:19.063257 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd" event={"ID":"09c6d33c-5374-4ad0-b00d-60c1ba64dec7","Type":"ContainerDied","Data":"cb1421af7203825d332013220b52ba05196acda1a2c828233236e3b0e4713834"} Oct 30 18:49:19.063440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:19.063321 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb1421af7203825d332013220b52ba05196acda1a2c828233236e3b0e4713834" Oct 30 18:49:19.870742 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:19.870706 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:49:22.223080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:22.223050 4862 scope.go:122] "RemoveContainer" containerID="c8bac789017b08201d9691aea8d86d9f1abf6ac63cc463d2cd9dd2e6ab2e98c0" Oct 30 18:49:23.754345 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.754316 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg"] Oct 30 18:49:23.754975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.754960 4862 cpu_manager.go:414] "containerMap: removing container" podUID="09c6d33c-5374-4ad0-b00d-60c1ba64dec7" containerName="revoke-keys" Oct 30 18:49:23.755021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.754977 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c6d33c-5374-4ad0-b00d-60c1ba64dec7" containerName="revoke-keys" Oct 30 18:49:23.755090 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.755081 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c6d33c-5374-4ad0-b00d-60c1ba64dec7" containerName="revoke-keys" Oct 30 18:49:23.760537 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.760516 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:23.778653 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.778631 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg"] Oct 30 18:49:23.786400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.786364 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:49:23.786667 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.786647 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:49:23.830174 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.830151 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgvs\" (UniqueName: \"kubernetes.io/projected/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-kube-api-access-lzgvs\") pod \"maas-api-revoke-sub-347a9bb0-m27tg\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:23.830333 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.830182 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-347a9bb0-m27tg\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:23.830333 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.830216 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-347a9bb0-m27tg\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:23.931369 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.931341 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgvs\" (UniqueName: \"kubernetes.io/projected/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-kube-api-access-lzgvs\") pod \"maas-api-revoke-sub-347a9bb0-m27tg\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:23.931480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.931373 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-347a9bb0-m27tg\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:23.931480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.931401 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-347a9bb0-m27tg\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:23.932105 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.932087 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-347a9bb0-m27tg\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:23.934257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.934232 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-347a9bb0-m27tg\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:23.951171 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:23.951150 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgvs\" (UniqueName: \"kubernetes.io/projected/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-kube-api-access-lzgvs\") pod \"maas-api-revoke-sub-347a9bb0-m27tg\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:24.074214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:24.074187 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:24.432125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:24.432095 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg"] Oct 30 18:49:24.432296 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:49:24.432135 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b6e9fb_aeaf_4661_8325_8b15cbfef68d.slice/crio-a082e996050cb5730de7230cee78ce4ac1772514767bdba590f5e075c9aad180 WatchSource:0}: Error finding container a082e996050cb5730de7230cee78ce4ac1772514767bdba590f5e075c9aad180: Status 404 returned error can't find the container with id a082e996050cb5730de7230cee78ce4ac1772514767bdba590f5e075c9aad180 Oct 30 18:49:25.097714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:25.097683 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" event={"ID":"92b6e9fb-aeaf-4661-8325-8b15cbfef68d","Type":"ContainerStarted","Data":"99f6560f52c26b05a89757a99c599b6cefe88c83882e83a87ef04c86d40b7d88"} Oct 30 18:49:25.098106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:25.097723 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" event={"ID":"92b6e9fb-aeaf-4661-8325-8b15cbfef68d","Type":"ContainerStarted","Data":"a082e996050cb5730de7230cee78ce4ac1772514767bdba590f5e075c9aad180"} Oct 30 18:49:25.136256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:25.136202 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" podStartSLOduration=2.136189214 podStartE2EDuration="2.136189214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:25.135186638 +0000 UTC m=+3064.184231544" watchObservedRunningTime="2026-10-30 18:49:25.136189214 +0000 UTC m=+3064.185234127" Oct 30 18:49:26.106602 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.106553 4862 generic.go:356] "Generic (PLEG): container finished" podID="92b6e9fb-aeaf-4661-8325-8b15cbfef68d" containerID="99f6560f52c26b05a89757a99c599b6cefe88c83882e83a87ef04c86d40b7d88" exitCode=0 Oct 30 18:49:26.107020 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.106640 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" event={"ID":"92b6e9fb-aeaf-4661-8325-8b15cbfef68d","Type":"ContainerDied","Data":"99f6560f52c26b05a89757a99c599b6cefe88c83882e83a87ef04c86d40b7d88"} Oct 30 18:49:26.512730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.512654 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74"] Oct 30 18:49:26.518169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.518148 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.534706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.534680 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74"] Oct 30 18:49:26.624397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.624357 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.624547 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.624485 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.624612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.624552 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7wbn\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-kube-api-access-g7wbn\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.624698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.624678 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.725776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.725746 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.725913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.725809 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.725913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.725865 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7wbn\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-kube-api-access-g7wbn\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.726007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.725929 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.726570 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.726548 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.726686 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.726621 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.728360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.728334 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.745491 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.745469 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7wbn\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-kube-api-access-g7wbn\") pod \"payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:26.833439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:26.833414 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:27.138221 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:49:27.138193 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d95431c_05b8_4d08_8024_278419fe34e4.slice/crio-3f77e464a4e48e1d57f5654d12ce8c63e984960dfa679145cf6f7306411c4b0e WatchSource:0}: Error finding container 3f77e464a4e48e1d57f5654d12ce8c63e984960dfa679145cf6f7306411c4b0e: Status 404 returned error can't find the container with id 3f77e464a4e48e1d57f5654d12ce8c63e984960dfa679145cf6f7306411c4b0e Oct 30 18:49:27.138593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.138480 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74"] Oct 30 18:49:27.266582 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.266556 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:27.335179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.333584 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-kube-api-access-lzgvs\" (UniqueName: \"kubernetes.io/projected/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-kube-api-access-lzgvs\") pod \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " Oct 30 18:49:27.335179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.333695 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-xks-ca\") pod \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " Oct 30 18:49:27.335179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.333742 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-openshift-ca\") pod \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\" (UID: \"92b6e9fb-aeaf-4661-8325-8b15cbfef68d\") " Oct 30 18:49:27.335179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.334655 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-openshift-ca" pod "92b6e9fb-aeaf-4661-8325-8b15cbfef68d" (UID: "92b6e9fb-aeaf-4661-8325-8b15cbfef68d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:27.340749 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.340720 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-xks-ca" pod "92b6e9fb-aeaf-4661-8325-8b15cbfef68d" (UID: "92b6e9fb-aeaf-4661-8325-8b15cbfef68d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:49:27.342249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.342220 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-kube-api-access-lzgvs" pod "92b6e9fb-aeaf-4661-8325-8b15cbfef68d" (UID: "92b6e9fb-aeaf-4661-8325-8b15cbfef68d"). InnerVolumeSpecName "kube-api-access-lzgvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:27.342401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.342284 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9"] Oct 30 18:49:27.350646 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.350616 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74"] Oct 30 18:49:27.435362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.435254 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:27.435362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.435311 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:27.435362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:27.435325 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzgvs\" (UniqueName: \"kubernetes.io/projected/92b6e9fb-aeaf-4661-8325-8b15cbfef68d-kube-api-access-lzgvs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:28.121462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:28.121437 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" Oct 30 18:49:28.121652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:28.121474 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg" event={"ID":"92b6e9fb-aeaf-4661-8325-8b15cbfef68d","Type":"ContainerDied","Data":"a082e996050cb5730de7230cee78ce4ac1772514767bdba590f5e075c9aad180"} Oct 30 18:49:28.121652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:28.121520 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a082e996050cb5730de7230cee78ce4ac1772514767bdba590f5e075c9aad180" Oct 30 18:49:28.123234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:28.123203 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" event={"ID":"3d95431c-05b8-4d08-8024-278419fe34e4","Type":"ContainerStarted","Data":"581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8"} Oct 30 18:49:28.123390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:28.123279 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" event={"ID":"3d95431c-05b8-4d08-8024-278419fe34e4","Type":"ContainerStarted","Data":"3f77e464a4e48e1d57f5654d12ce8c63e984960dfa679145cf6f7306411c4b0e"} Oct 30 18:49:28.123435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:28.123423 4862 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" secret="" err="secret \"payload-processing-external-model-e2e--541faec9-dockercfg-76bvw\" not found" Oct 30 18:49:28.123475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:28.123440 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:28.164295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:28.164234 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" podStartSLOduration=2.164222889 podStartE2EDuration="2.164222889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:28.16284373 +0000 UTC m=+3067.211888643" watchObservedRunningTime="2026-10-30 18:49:28.164222889 +0000 UTC m=+3067.213267802" Oct 30 18:49:28.253383 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.253355 4862 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-plugins-e2e-extmod-a-d1a330: configmap "payload-processing-external-model-plugins-e2e-extmod-a-d1a330" not found Oct 30 18:49:28.253560 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.253394 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d1a330/agc-cred-b58a5af19216: secret "agc-cred-b58a5af19216" not found Oct 30 18:49:28.253560 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.253414 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74: secret "agc-cred-b58a5af19216" not found Oct 30 18:49:28.253560 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.253452 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume podName:3d95431c-05b8-4d08-8024-278419fe34e4 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:28.75342914 +0000 UTC m=+3067.802474034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-d1a330" not found Oct 30 18:49:28.253560 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.253493 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials podName:3d95431c-05b8-4d08-8024-278419fe34e4 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:28.753475042 +0000 UTC m=+3067.802519933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4") : secret "agc-cred-b58a5af19216" not found Oct 30 18:49:28.759624 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.759594 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d1a330/agc-cred-b58a5af19216: secret "agc-cred-b58a5af19216" not found Oct 30 18:49:28.759624 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.759616 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74: secret "agc-cred-b58a5af19216" not found Oct 30 18:49:28.759855 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.759658 4862 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-plugins-e2e-extmod-a-d1a330: configmap "payload-processing-external-model-plugins-e2e-extmod-a-d1a330" not found Oct 30 18:49:28.759855 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.759676 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials podName:3d95431c-05b8-4d08-8024-278419fe34e4 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:29.759660661 +0000 UTC m=+3068.808705559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4") : secret "agc-cred-b58a5af19216" not found Oct 30 18:49:28.759855 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:28.759711 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume podName:3d95431c-05b8-4d08-8024-278419fe34e4 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:29.759699152 +0000 UTC m=+3068.808744065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-d1a330" not found Oct 30 18:49:29.772463 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:29.772432 4862 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-plugins-e2e-extmod-a-d1a330: configmap "payload-processing-external-model-plugins-e2e-extmod-a-d1a330" not found Oct 30 18:49:29.772848 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:29.772521 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume podName:3d95431c-05b8-4d08-8024-278419fe34e4 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:31.772508451 +0000 UTC m=+3070.821553360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-d1a330" not found Oct 30 18:49:29.772848 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:29.772532 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d1a330/agc-cred-b58a5af19216: secret "agc-cred-b58a5af19216" not found Oct 30 18:49:29.772848 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:29.772548 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74: secret "agc-cred-b58a5af19216" not found Oct 30 18:49:29.772848 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:29.772600 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials podName:3d95431c-05b8-4d08-8024-278419fe34e4 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:31.772586683 +0000 UTC m=+3070.821631597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4") : secret "agc-cred-b58a5af19216" not found Oct 30 18:49:31.851399 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:31.851363 4862 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-plugins-e2e-extmod-a-d1a330: configmap "payload-processing-external-model-plugins-e2e-extmod-a-d1a330" not found Oct 30 18:49:31.851813 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:31.851430 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume podName:3d95431c-05b8-4d08-8024-278419fe34e4 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:35.851416689 +0000 UTC m=+3074.900461593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-d1a330" not found Oct 30 18:49:31.851813 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:31.851451 4862 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d1a330/agc-cred-b58a5af19216: secret "agc-cred-b58a5af19216" not found Oct 30 18:49:31.851813 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:31.851468 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74: secret "agc-cred-b58a5af19216" not found Oct 30 18:49:31.851813 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:31.851512 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials podName:3d95431c-05b8-4d08-8024-278419fe34e4 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:35.8514985 +0000 UTC m=+3074.900543412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4") : secret "agc-cred-b58a5af19216" not found Oct 30 18:49:32.343383 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.343334 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" podUID="8bf01d42-b18a-4984-af8a-9c125c388775" containerName="payload-processing" containerID="cri-o://a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8" gracePeriod=25 Oct 30 18:49:32.624348 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.624324 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:49:32.764172 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.764137 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-plugins-config-volume\") pod \"8bf01d42-b18a-4984-af8a-9c125c388775\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " Oct 30 18:49:32.764375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.764315 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-provider-credentials\") pod \"8bf01d42-b18a-4984-af8a-9c125c388775\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " Oct 30 18:49:32.764375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.764347 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-routing-overlay-volume\") pod \"8bf01d42-b18a-4984-af8a-9c125c388775\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " Oct 30 18:49:32.764556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.764530 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-plugins-config-volume" pod "8bf01d42-b18a-4984-af8a-9c125c388775" (UID: "8bf01d42-b18a-4984-af8a-9c125c388775"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:32.764628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.764537 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-kube-api-access-ppxrn\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-kube-api-access-ppxrn\") pod \"8bf01d42-b18a-4984-af8a-9c125c388775\" (UID: \"8bf01d42-b18a-4984-af8a-9c125c388775\") " Oct 30 18:49:32.764768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.764743 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-routing-overlay-volume" pod "8bf01d42-b18a-4984-af8a-9c125c388775" (UID: "8bf01d42-b18a-4984-af8a-9c125c388775"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:32.765209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.765183 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:32.765209 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.765207 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8bf01d42-b18a-4984-af8a-9c125c388775-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:32.766699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.766678 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-kube-api-access-ppxrn" pod "8bf01d42-b18a-4984-af8a-9c125c388775" (UID: "8bf01d42-b18a-4984-af8a-9c125c388775"). InnerVolumeSpecName "kube-api-access-ppxrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:32.766765 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.766700 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-provider-credentials" pod "8bf01d42-b18a-4984-af8a-9c125c388775" (UID: "8bf01d42-b18a-4984-af8a-9c125c388775"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:32.866804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.866718 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:32.866804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:32.866746 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppxrn\" (UniqueName: \"kubernetes.io/projected/8bf01d42-b18a-4984-af8a-9c125c388775-kube-api-access-ppxrn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:33.163358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:33.163281 4862 generic.go:356] "Generic (PLEG): container finished" podID="8bf01d42-b18a-4984-af8a-9c125c388775" containerID="a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8" exitCode=0 Oct 30 18:49:33.163358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:33.163335 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" Oct 30 18:49:33.163563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:33.163370 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" event={"ID":"8bf01d42-b18a-4984-af8a-9c125c388775","Type":"ContainerDied","Data":"a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8"} Oct 30 18:49:33.163563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:33.163422 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9" event={"ID":"8bf01d42-b18a-4984-af8a-9c125c388775","Type":"ContainerDied","Data":"3b1fa19b82368339ee3a3d0722a5ea53b9aa3e54e7defad2808899fdc00b7b4b"} Oct 30 18:49:33.163563 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:33.163446 4862 scope.go:122] "RemoveContainer" containerID="a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8" Oct 30 18:49:33.173692 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:33.173677 4862 scope.go:122] "RemoveContainer" containerID="a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8" Oct 30 18:49:33.173974 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:33.173951 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8\": container with ID starting with a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8 not found: ID does not exist" containerID="a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8" Oct 30 18:49:33.174045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:33.173982 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8"} err="failed to get container status \"a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8\": rpc error: code = NotFound desc = could not find container \"a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8\": container with ID starting with a806f9c17e4f6bb330df2c1efdf2974bcbc8fbe7f79ce8bff15437f0a19152d8 not found: ID does not exist" Oct 30 18:49:33.218447 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:33.218421 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9"] Oct 30 18:49:33.222175 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:33.222136 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-d65brs4p9"] Oct 30 18:49:34.130853 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.130816 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" podUID="3d95431c-05b8-4d08-8024-278419fe34e4" containerName="payload-processing" containerID="cri-o://581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8" gracePeriod=25 Oct 30 18:49:34.509613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.509590 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:34.590795 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.590769 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-kube-api-access-g7wbn\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-kube-api-access-g7wbn\") pod \"3d95431c-05b8-4d08-8024-278419fe34e4\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " Oct 30 18:49:34.590961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.590909 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials\") pod \"3d95431c-05b8-4d08-8024-278419fe34e4\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " Oct 30 18:49:34.591025 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.591008 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-routing-overlay-volume\") pod \"3d95431c-05b8-4d08-8024-278419fe34e4\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " Oct 30 18:49:34.591109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.591056 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume\") pod \"3d95431c-05b8-4d08-8024-278419fe34e4\" (UID: \"3d95431c-05b8-4d08-8024-278419fe34e4\") " Oct 30 18:49:34.591396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.591365 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-routing-overlay-volume" pod "3d95431c-05b8-4d08-8024-278419fe34e4" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:34.591627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.591590 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:34.591816 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.591792 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume" pod "3d95431c-05b8-4d08-8024-278419fe34e4" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:34.593195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.593170 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-kube-api-access-g7wbn" pod "3d95431c-05b8-4d08-8024-278419fe34e4" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4"). InnerVolumeSpecName "kube-api-access-g7wbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:34.593599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.593575 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials" pod "3d95431c-05b8-4d08-8024-278419fe34e4" (UID: "3d95431c-05b8-4d08-8024-278419fe34e4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:34.693081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.693025 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7wbn\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-kube-api-access-g7wbn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:34.693081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.693049 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d95431c-05b8-4d08-8024-278419fe34e4-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:34.693081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:34.693058 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d95431c-05b8-4d08-8024-278419fe34e4-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:35.063697 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.063668 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf01d42-b18a-4984-af8a-9c125c388775" path="/var/lib/kubelet/pods/8bf01d42-b18a-4984-af8a-9c125c388775/volumes" Oct 30 18:49:35.179400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.179369 4862 generic.go:356] "Generic (PLEG): container finished" podID="3d95431c-05b8-4d08-8024-278419fe34e4" containerID="581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8" exitCode=0 Oct 30 18:49:35.179781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.179412 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" Oct 30 18:49:35.179781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.179454 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" event={"ID":"3d95431c-05b8-4d08-8024-278419fe34e4","Type":"ContainerDied","Data":"581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8"} Oct 30 18:49:35.179781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.179503 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74" event={"ID":"3d95431c-05b8-4d08-8024-278419fe34e4","Type":"ContainerDied","Data":"3f77e464a4e48e1d57f5654d12ce8c63e984960dfa679145cf6f7306411c4b0e"} Oct 30 18:49:35.179781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.179533 4862 scope.go:122] "RemoveContainer" containerID="581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8" Oct 30 18:49:35.189815 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.189794 4862 scope.go:122] "RemoveContainer" containerID="581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8" Oct 30 18:49:35.190078 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:35.190058 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8\": container with ID starting with 581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8 not found: ID does not exist" containerID="581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8" Oct 30 18:49:35.190135 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.190087 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8"} err="failed to get container status \"581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8\": rpc error: code = NotFound desc = could not find container \"581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8\": container with ID starting with 581e33aefd9586c94166c43cf28f577de1ffce4a07321fadf26e672aada41cf8 not found: ID does not exist" Oct 30 18:49:35.234990 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.234971 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74"] Oct 30 18:49:35.247021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.247000 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d1a330/payload-processing-external-model-e2e-extmod-a-d1a330-79646pz74"] Oct 30 18:49:35.250134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:35.250119 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4"] Oct 30 18:49:37.066402 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:37.066366 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d95431c-05b8-4d08-8024-278419fe34e4" path="/var/lib/kubelet/pods/3d95431c-05b8-4d08-8024-278419fe34e4/volumes" Oct 30 18:49:38.627521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.627492 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw"] Oct 30 18:49:38.628172 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.628155 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8bf01d42-b18a-4984-af8a-9c125c388775" containerName="payload-processing" Oct 30 18:49:38.628172 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.628173 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf01d42-b18a-4984-af8a-9c125c388775" containerName="payload-processing" Oct 30 18:49:38.628343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.628199 4862 cpu_manager.go:414] "containerMap: removing container" podUID="92b6e9fb-aeaf-4661-8325-8b15cbfef68d" containerName="revoke-keys" Oct 30 18:49:38.628343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.628204 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b6e9fb-aeaf-4661-8325-8b15cbfef68d" containerName="revoke-keys" Oct 30 18:49:38.628343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.628219 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3d95431c-05b8-4d08-8024-278419fe34e4" containerName="payload-processing" Oct 30 18:49:38.628343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.628224 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d95431c-05b8-4d08-8024-278419fe34e4" containerName="payload-processing" Oct 30 18:49:38.628536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.628349 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf01d42-b18a-4984-af8a-9c125c388775" containerName="payload-processing" Oct 30 18:49:38.628536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.628361 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b6e9fb-aeaf-4661-8325-8b15cbfef68d" containerName="revoke-keys" Oct 30 18:49:38.628536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.628370 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d95431c-05b8-4d08-8024-278419fe34e4" containerName="payload-processing" Oct 30 18:49:38.636161 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.636138 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.649542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.649515 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw"] Oct 30 18:49:38.657066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.657036 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 18:49:38.657308 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.657239 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 18:49:38.657414 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.657348 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 18:49:38.657414 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.657404 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-940f10a431a6\"" Oct 30 18:49:38.657656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.657637 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-j4vsz\"" Oct 30 18:49:38.663920 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.663900 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 18:49:38.802293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.799503 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-routing-overlay-volume\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.802293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.799626 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.802293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.799707 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.802293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.799805 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthdr\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.843283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.840306 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw"] Oct 30 18:49:38.843283 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:38.840845 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sthdr plugins-config-volume provider-credentials routing-overlay-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" podUID="8f4c7639-f40a-445b-b854-d583bc7386d2" Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.902156 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.902249 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthdr\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.902438 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-routing-overlay-volume\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.902518 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:38.902631 4862 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:38.902703 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume podName:8f4c7639-f40a-445b-b854-d583bc7386d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:39.402681713 +0000 UTC m=+3078.451726607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume") pod "payload-processing-external-model-595fbb7bb9-vgwvw" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2") : configmap "payload-processing-external-model-plugins" not found Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:38.903000 4862 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-940f10a431a6: secret "agc-cred-940f10a431a6" not found Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:38.903014 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw: secret "agc-cred-940f10a431a6" not found Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:38.903055 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials podName:8f4c7639-f40a-445b-b854-d583bc7386d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:39.40304198 +0000 UTC m=+3078.452086874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials") pod "payload-processing-external-model-595fbb7bb9-vgwvw" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2") : secret "agc-cred-940f10a431a6" not found Oct 30 18:49:38.904342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:38.903918 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-routing-overlay-volume\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:38.936013 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:38.933866 4862 projected.go:196] Error preparing data for projected volume kube-api-access-sthdr for pod models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 18:49:38.936013 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:38.933942 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr podName:8f4c7639-f40a-445b-b854-d583bc7386d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:39.433924045 +0000 UTC m=+3078.482968950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sthdr" (UniqueName: "kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr") pod "payload-processing-external-model-595fbb7bb9-vgwvw" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 18:49:39.243340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:39.242361 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:39.256820 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:39.253713 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:39.324304 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:39.324050 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-routing-overlay-volume\") pod \"8f4c7639-f40a-445b-b854-d583bc7386d2\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " Oct 30 18:49:39.324508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:39.324412 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-routing-overlay-volume" pod "8f4c7639-f40a-445b-b854-d583bc7386d2" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:39.324843 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:39.324819 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:39.435293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:39.427695 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:39.435640 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:39.435614 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:39.435880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:39.435859 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthdr\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:39.436344 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:39.434398 4862 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 18:49:39.436489 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:39.436422 4862 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-940f10a431a6: secret "agc-cred-940f10a431a6" not found Oct 30 18:49:39.436489 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:39.436444 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw: secret "agc-cred-940f10a431a6" not found Oct 30 18:49:39.436489 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:39.436422 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume podName:8f4c7639-f40a-445b-b854-d583bc7386d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:40.436397197 +0000 UTC m=+3079.485442102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume") pod "payload-processing-external-model-595fbb7bb9-vgwvw" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2") : configmap "payload-processing-external-model-plugins" not found Oct 30 18:49:39.436645 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:39.436509 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials podName:8f4c7639-f40a-445b-b854-d583bc7386d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:40.436492124 +0000 UTC m=+3079.485537034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials") pod "payload-processing-external-model-595fbb7bb9-vgwvw" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2") : secret "agc-cred-940f10a431a6" not found Oct 30 18:49:39.452343 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:39.452317 4862 projected.go:196] Error preparing data for projected volume kube-api-access-sthdr for pod models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 18:49:39.452472 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:39.452388 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr podName:8f4c7639-f40a-445b-b854-d583bc7386d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:40.452372766 +0000 UTC m=+3079.501417672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sthdr" (UniqueName: "kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr") pod "payload-processing-external-model-595fbb7bb9-vgwvw" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 18:49:40.252287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.251840 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" podUID="a5f77ec5-bd2d-4914-91a5-9d7a588771af" containerName="payload-processing" containerID="cri-o://6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b" gracePeriod=25 Oct 30 18:49:40.258303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.253734 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:40.424480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.422406 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw"] Oct 30 18:49:40.431213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.431182 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw"] Oct 30 18:49:40.527567 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.527461 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:40.528115 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.528063 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:40.528492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.528471 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthdr\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr\") pod \"payload-processing-external-model-595fbb7bb9-vgwvw\" (UID: \"8f4c7639-f40a-445b-b854-d583bc7386d2\") " pod="models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw" Oct 30 18:49:40.529193 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:40.529172 4862 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: object "models-as-a-service"/"payload-processing-external-model-plugins" not registered Oct 30 18:49:40.529407 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:40.529391 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume podName:8f4c7639-f40a-445b-b854-d583bc7386d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:42.529369401 +0000 UTC m=+3081.578414311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume") pod "payload-processing-external-model-595fbb7bb9-vgwvw" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2") : object "models-as-a-service"/"payload-processing-external-model-plugins" not registered Oct 30 18:49:40.529789 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:40.529771 4862 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-940f10a431a6: object "models-as-a-service"/"agc-cred-940f10a431a6" not registered Oct 30 18:49:40.529919 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:40.529904 4862 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw: object "models-as-a-service"/"agc-cred-940f10a431a6" not registered Oct 30 18:49:40.530061 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:40.530048 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials podName:8f4c7639-f40a-445b-b854-d583bc7386d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:42.530031785 +0000 UTC m=+3081.579076690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials") pod "payload-processing-external-model-595fbb7bb9-vgwvw" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2") : object "models-as-a-service"/"agc-cred-940f10a431a6" not registered Oct 30 18:49:40.560963 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:40.555004 4862 projected.go:291] Couldn't get configMap models-as-a-service/kube-root-ca.crt: object "models-as-a-service"/"kube-root-ca.crt" not registered Oct 30 18:49:40.560963 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:40.555033 4862 projected.go:291] Couldn't get configMap models-as-a-service/openshift-service-ca.crt: object "models-as-a-service"/"openshift-service-ca.crt" not registered Oct 30 18:49:40.560963 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:40.555049 4862 projected.go:196] Error preparing data for projected volume kube-api-access-sthdr for pod models-as-a-service/payload-processing-external-model-595fbb7bb9-vgwvw: [failed to fetch token: serviceaccounts "payload-processing-external-model" is forbidden: User "system:node:ip-10-0-128-246.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "models-as-a-service": no relationship found between node 'ip-10-0-128-246.ec2.internal' and this object, object "models-as-a-service"/"kube-root-ca.crt" not registered, object "models-as-a-service"/"openshift-service-ca.crt" not registered] Oct 30 18:49:40.560963 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:40.555116 4862 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr podName:8f4c7639-f40a-445b-b854-d583bc7386d2 nodeName:}" failed. No retries permitted until 2026-10-30 18:49:42.555098131 +0000 UTC m=+3081.604143038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sthdr" (UniqueName: "kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr") pod "payload-processing-external-model-595fbb7bb9-vgwvw" (UID: "8f4c7639-f40a-445b-b854-d583bc7386d2") : [failed to fetch token: serviceaccounts "payload-processing-external-model" is forbidden: User "system:node:ip-10-0-128-246.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "models-as-a-service": no relationship found between node 'ip-10-0-128-246.ec2.internal' and this object, object "models-as-a-service"/"kube-root-ca.crt" not registered, object "models-as-a-service"/"openshift-service-ca.crt" not registered] Oct 30 18:49:40.652599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.652560 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:40.652599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.652601 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sthdr\" (UniqueName: \"kubernetes.io/projected/8f4c7639-f40a-445b-b854-d583bc7386d2-kube-api-access-sthdr\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:40.652846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.652617 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c7639-f40a-445b-b854-d583bc7386d2-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:40.738808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.738462 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:49:40.754149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.754117 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-plugins-config-volume\") pod \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " Oct 30 18:49:40.754909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.754708 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-plugins-config-volume" pod "a5f77ec5-bd2d-4914-91a5-9d7a588771af" (UID: "a5f77ec5-bd2d-4914-91a5-9d7a588771af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:40.755097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.755077 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-provider-credentials\") pod \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " Oct 30 18:49:40.755911 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.755890 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-routing-overlay-volume\") pod \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " Oct 30 18:49:40.756739 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.756597 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-routing-overlay-volume" pod "a5f77ec5-bd2d-4914-91a5-9d7a588771af" (UID: "a5f77ec5-bd2d-4914-91a5-9d7a588771af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:49:40.757167 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.757146 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-kube-api-access-qfv97\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-kube-api-access-qfv97\") pod \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\" (UID: \"a5f77ec5-bd2d-4914-91a5-9d7a588771af\") " Oct 30 18:49:40.760312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.760288 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:40.760607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.760539 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f77ec5-bd2d-4914-91a5-9d7a588771af-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:40.765827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.765389 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-provider-credentials" pod "a5f77ec5-bd2d-4914-91a5-9d7a588771af" (UID: "a5f77ec5-bd2d-4914-91a5-9d7a588771af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:40.765827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.765779 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-kube-api-access-qfv97" pod "a5f77ec5-bd2d-4914-91a5-9d7a588771af" (UID: "a5f77ec5-bd2d-4914-91a5-9d7a588771af"). InnerVolumeSpecName "kube-api-access-qfv97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:49:40.896186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.895037 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfv97\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-kube-api-access-qfv97\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:40.896186 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:40.895073 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f77ec5-bd2d-4914-91a5-9d7a588771af-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:49:41.071664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.071621 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f4c7639-f40a-445b-b854-d583bc7386d2" path="/var/lib/kubelet/pods/8f4c7639-f40a-445b-b854-d583bc7386d2/volumes" Oct 30 18:49:41.277430 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.277339 4862 generic.go:356] "Generic (PLEG): container finished" podID="a5f77ec5-bd2d-4914-91a5-9d7a588771af" containerID="6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b" exitCode=0 Oct 30 18:49:41.277941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.277436 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" event={"ID":"a5f77ec5-bd2d-4914-91a5-9d7a588771af","Type":"ContainerDied","Data":"6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b"} Oct 30 18:49:41.277941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.277485 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" event={"ID":"a5f77ec5-bd2d-4914-91a5-9d7a588771af","Type":"ContainerDied","Data":"45791cfdf2470900d0289190394bfd9c67afdfd4f1e2620add95664b5da8df81"} Oct 30 18:49:41.277941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.277516 4862 scope.go:122] "RemoveContainer" containerID="6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b" Oct 30 18:49:41.277941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.277672 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4" Oct 30 18:49:41.305411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.305391 4862 scope.go:122] "RemoveContainer" containerID="6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b" Oct 30 18:49:41.305690 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:41.305672 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b\": container with ID starting with 6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b not found: ID does not exist" containerID="6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b" Oct 30 18:49:41.305772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.305701 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b"} err="failed to get container status \"6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b\": rpc error: code = NotFound desc = could not find container \"6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b\": container with ID starting with 6d412f25f729d42a07e56ccaa88d6d51afe36e86029d1a7b2932b655f8db261b not found: ID does not exist" Oct 30 18:49:41.336666 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.336636 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4"] Oct 30 18:49:41.345909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:41.345873 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1c4bcd/payload-processing-external-model-e2e-extmod-a-1c4bcd-65d9hspl4"] Oct 30 18:49:42.596488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.596460 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2"] Oct 30 18:49:42.597110 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.597089 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a5f77ec5-bd2d-4914-91a5-9d7a588771af" containerName="payload-processing" Oct 30 18:49:42.597157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.597111 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f77ec5-bd2d-4914-91a5-9d7a588771af" containerName="payload-processing" Oct 30 18:49:42.597358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.597252 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f77ec5-bd2d-4914-91a5-9d7a588771af" containerName="payload-processing" Oct 30 18:49:42.602147 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.602126 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.620206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.620176 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-credential-socket\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.620434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.620252 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-socket\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.620434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.620344 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-envoy\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.620434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.620399 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e97e48f-6b9d-4832-9876-18c1a03d431a-istiod-ca-cert\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.620434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.620426 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-certs\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.620670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.620473 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-podinfo\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.620670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.620502 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-data\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.620670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.620539 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dckct\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-kube-api-access-dckct\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.620670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.620638 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-token\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.623465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.621999 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1aa59a-openshift-default-dockercfg-76g2m\"" Oct 30 18:49:42.644302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.640471 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2"] Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.759540 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-token\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.759707 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-credential-socket\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.759781 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-socket\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.759855 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-envoy\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.759920 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e97e48f-6b9d-4832-9876-18c1a03d431a-istiod-ca-cert\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.759951 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-certs\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.760002 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-podinfo\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.760034 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-data\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.760076 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dckct\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-kube-api-access-dckct\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.761348 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-credential-socket\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.761604 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-socket\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.762121 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-certs\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.768288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.762737 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e97e48f-6b9d-4832-9876-18c1a03d431a-istiod-ca-cert\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.790286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.789219 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-data\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.807291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.804787 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-token\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.807291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.805443 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-podinfo\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.822329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.821380 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-envoy\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.847994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.832188 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dckct\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-kube-api-access-dckct\") pod \"e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:42.918924 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:42.918881 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:43.086799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:43.086743 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f77ec5-bd2d-4914-91a5-9d7a588771af" path="/var/lib/kubelet/pods/a5f77ec5-bd2d-4914-91a5-9d7a588771af/volumes" Oct 30 18:49:43.294815 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:49:43.294790 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e97e48f_6b9d_4832_9876_18c1a03d431a.slice/crio-02e856d4441fbd00f87706c51aa99aa07316233171b185b7fdc10298a88c0e6e WatchSource:0}: Error finding container 02e856d4441fbd00f87706c51aa99aa07316233171b185b7fdc10298a88c0e6e: Status 404 returned error can't find the container with id 02e856d4441fbd00f87706c51aa99aa07316233171b185b7fdc10298a88c0e6e Oct 30 18:49:43.295082 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:43.295061 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2"] Oct 30 18:49:43.297557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:43.297526 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:49:43.297676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:43.297586 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:49:43.297676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:43.297616 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:49:44.303248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:44.303211 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" event={"ID":"4e97e48f-6b9d-4832-9876-18c1a03d431a","Type":"ContainerStarted","Data":"d1a9c0e4430e76c95d953e88a5d8cbcfcfa6c48dd84c1555262c2f1ed8c61b10"} Oct 30 18:49:44.303248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:44.303277 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" event={"ID":"4e97e48f-6b9d-4832-9876-18c1a03d431a","Type":"ContainerStarted","Data":"02e856d4441fbd00f87706c51aa99aa07316233171b185b7fdc10298a88c0e6e"} Oct 30 18:49:44.406345 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:44.393250 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" podStartSLOduration=2.393229444 podStartE2EDuration="2.393229444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:44.375834489 +0000 UTC m=+3083.424879402" watchObservedRunningTime="2026-10-30 18:49:44.393229444 +0000 UTC m=+3083.442274383" Oct 30 18:49:44.919455 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:44.919425 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:44.925521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:44.925500 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:45.310693 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:45.310662 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:45.311651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:45.311627 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:49:48.035752 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:48.035723 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4"] Oct 30 18:49:48.048917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:48.048882 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94a96e4e-48jc4"] Oct 30 18:49:48.965849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:48.956714 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt"] Oct 30 18:49:48.968145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:48.968114 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt"] Oct 30 18:49:48.968620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:48.968333 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:49:48.984317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:48.984289 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1aa59a\"" Oct 30 18:49:48.998474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:48.998441 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1aa59a-dockercfg-gbmbw\"" Oct 30 18:49:49.011314 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:49.011247 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d21690-359c-495d-b82c-fa0e5a396e36-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt\" (UID: \"68d21690-359c-495d-b82c-fa0e5a396e36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:49:49.011515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:49.011490 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xv6c\" (UniqueName: \"kubernetes.io/projected/68d21690-359c-495d-b82c-fa0e5a396e36-kube-api-access-7xv6c\") pod \"payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt\" (UID: \"68d21690-359c-495d-b82c-fa0e5a396e36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:49:49.064007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:49.063978 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea" path="/var/lib/kubelet/pods/3398ceb5-ee5d-4d8e-b7cf-0c7af5cf44ea/volumes" Oct 30 18:49:49.113025 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:49.112994 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xv6c\" (UniqueName: \"kubernetes.io/projected/68d21690-359c-495d-b82c-fa0e5a396e36-kube-api-access-7xv6c\") pod \"payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt\" (UID: \"68d21690-359c-495d-b82c-fa0e5a396e36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:49:49.113183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:49.113150 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d21690-359c-495d-b82c-fa0e5a396e36-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt\" (UID: \"68d21690-359c-495d-b82c-fa0e5a396e36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:49:49.113858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:49.113833 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d21690-359c-495d-b82c-fa0e5a396e36-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt\" (UID: \"68d21690-359c-495d-b82c-fa0e5a396e36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:49:49.133488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:49.133463 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xv6c\" (UniqueName: \"kubernetes.io/projected/68d21690-359c-495d-b82c-fa0e5a396e36-kube-api-access-7xv6c\") pod \"payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt\" (UID: \"68d21690-359c-495d-b82c-fa0e5a396e36\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:49:49.285754 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:49.285679 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:49:49.592558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:49.592533 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt"] Oct 30 18:49:49.592718 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:49:49.592697 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d21690_359c_495d_b82c_fa0e5a396e36.slice/crio-99ec11ac1e4f3a4e0b874e5d65853c0b6e8999c97ee00f412abe7a20303ab5a8 WatchSource:0}: Error finding container 99ec11ac1e4f3a4e0b874e5d65853c0b6e8999c97ee00f412abe7a20303ab5a8: Status 404 returned error can't find the container with id 99ec11ac1e4f3a4e0b874e5d65853c0b6e8999c97ee00f412abe7a20303ab5a8 Oct 30 18:49:49.869078 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:49:49.869000 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:49:50.349420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:50.349385 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" event={"ID":"68d21690-359c-495d-b82c-fa0e5a396e36","Type":"ContainerStarted","Data":"d3f42c033dce73b74b5b32cc0247979a9e8a621d9b9474a428fb33ece437d8cb"} Oct 30 18:49:50.349907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:50.349441 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" event={"ID":"68d21690-359c-495d-b82c-fa0e5a396e36","Type":"ContainerStarted","Data":"99ec11ac1e4f3a4e0b874e5d65853c0b6e8999c97ee00f412abe7a20303ab5a8"} Oct 30 18:49:50.349907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:50.349480 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:49:50.388394 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:50.388351 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" podStartSLOduration=2.388335401 podStartE2EDuration="2.388335401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:50.387782955 +0000 UTC m=+3089.436827868" watchObservedRunningTime="2026-10-30 18:49:50.388335401 +0000 UTC m=+3089.437380317" Oct 30 18:49:53.415332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.415302 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr"] Oct 30 18:49:53.419829 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.419806 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.438688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.438665 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr"] Oct 30 18:49:53.448304 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.447386 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 18:49:53.448304 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.447417 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wlmz5\"" Oct 30 18:49:53.448304 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.447446 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 18:49:53.448304 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.447513 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 18:49:53.448663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.448646 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 18:49:53.448755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.448673 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-56887c927732\"" Oct 30 18:49:53.490888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.490867 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-routing-overlay-volume\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.491001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.490898 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-provider-credentials\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.491097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.491078 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6njj\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-kube-api-access-p6njj\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.491136 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.491108 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-plugins-config-volume\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.601285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.598399 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6njj\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-kube-api-access-p6njj\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.601285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.598454 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-plugins-config-volume\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.601285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.598645 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-routing-overlay-volume\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.601285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.598681 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-provider-credentials\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.601285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.600382 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-plugins-config-volume\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.601707 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.601298 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-routing-overlay-volume\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.604629 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.604604 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-provider-credentials\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.627943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.627911 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6njj\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-kube-api-access-p6njj\") pod \"payload-processing-external-model-6656cd76c4-lf7dr\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:53.745331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:53.735297 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:54.062255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:54.062227 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr"] Oct 30 18:49:54.063059 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:49:54.063032 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9caef399_ed47_4777_9215_df9b20da7242.slice/crio-32aa2f4d39c8cd9dd0afa4f7441b2c39ca10f8be20d6b2125fc9f95a34ca3779 WatchSource:0}: Error finding container 32aa2f4d39c8cd9dd0afa4f7441b2c39ca10f8be20d6b2125fc9f95a34ca3779: Status 404 returned error can't find the container with id 32aa2f4d39c8cd9dd0afa4f7441b2c39ca10f8be20d6b2125fc9f95a34ca3779 Oct 30 18:49:54.385741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:54.385371 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" event={"ID":"9caef399-ed47-4777-9215-df9b20da7242","Type":"ContainerStarted","Data":"c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28"} Oct 30 18:49:54.385741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:54.385450 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:49:54.385741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:54.385478 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" event={"ID":"9caef399-ed47-4777-9215-df9b20da7242","Type":"ContainerStarted","Data":"32aa2f4d39c8cd9dd0afa4f7441b2c39ca10f8be20d6b2125fc9f95a34ca3779"} Oct 30 18:49:54.427468 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:54.427422 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" podStartSLOduration=1.4274077059999999 podStartE2EDuration="1.427407706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:54.424797817 +0000 UTC m=+3093.473842729" watchObservedRunningTime="2026-10-30 18:49:54.427407706 +0000 UTC m=+3093.476452618" Oct 30 18:49:57.183286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:49:57.183129 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:50:01.359908 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:01.359830 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 18:50:05.396167 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:05.396137 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:50:19.871091 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:50:19.871053 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:50:28.031436 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:28.031404 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc"] Oct 30 18:50:28.044021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:28.043997 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cdb2c67-tdprc"] Oct 30 18:50:29.142711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:29.142665 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05" path="/var/lib/kubelet/pods/a0bb4ab2-9038-4a6d-a3e9-cf2a5c25ff05/volumes" Oct 30 18:50:35.762290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.752605 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8"] Oct 30 18:50:35.840712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.839558 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:35.841573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.841548 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8"] Oct 30 18:50:35.841782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.841767 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn"] Oct 30 18:50:35.841912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.841900 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d128ee44-xfwtn"] Oct 30 18:50:35.842034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.842022 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7"] Oct 30 18:50:35.869448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.869421 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7"] Oct 30 18:50:35.869710 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.869691 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:35.903286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.886287 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c52ec7-dockercfg-jl2rw\"" Oct 30 18:50:35.903286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.886356 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c52ec7\"" Oct 30 18:50:35.944044 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.944013 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpzj\" (UniqueName: \"kubernetes.io/projected/abd590d7-b8b0-4356-9566-c3e62499c87b-kube-api-access-kbpzj\") pod \"payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8\" (UID: \"abd590d7-b8b0-4356-9566-c3e62499c87b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:35.944395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.944369 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqddf\" (UniqueName: \"kubernetes.io/projected/6e82685b-9c65-432b-a863-c39edd6950e4-kube-api-access-jqddf\") pod \"payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7\" (UID: \"6e82685b-9c65-432b-a863-c39edd6950e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:35.944572 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.944556 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd590d7-b8b0-4356-9566-c3e62499c87b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8\" (UID: \"abd590d7-b8b0-4356-9566-c3e62499c87b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:35.945808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:35.945784 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e82685b-9c65-432b-a863-c39edd6950e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7\" (UID: \"6e82685b-9c65-432b-a863-c39edd6950e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:36.060332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.047128 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e82685b-9c65-432b-a863-c39edd6950e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7\" (UID: \"6e82685b-9c65-432b-a863-c39edd6950e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:36.060332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.047322 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpzj\" (UniqueName: \"kubernetes.io/projected/abd590d7-b8b0-4356-9566-c3e62499c87b-kube-api-access-kbpzj\") pod \"payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8\" (UID: \"abd590d7-b8b0-4356-9566-c3e62499c87b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:36.060332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.047440 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqddf\" (UniqueName: \"kubernetes.io/projected/6e82685b-9c65-432b-a863-c39edd6950e4-kube-api-access-jqddf\") pod \"payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7\" (UID: \"6e82685b-9c65-432b-a863-c39edd6950e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:36.060332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.047484 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd590d7-b8b0-4356-9566-c3e62499c87b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8\" (UID: \"abd590d7-b8b0-4356-9566-c3e62499c87b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:36.060332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.048436 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd590d7-b8b0-4356-9566-c3e62499c87b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8\" (UID: \"abd590d7-b8b0-4356-9566-c3e62499c87b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:36.060332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.049213 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e82685b-9c65-432b-a863-c39edd6950e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7\" (UID: \"6e82685b-9c65-432b-a863-c39edd6950e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:36.103994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.103958 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqddf\" (UniqueName: \"kubernetes.io/projected/6e82685b-9c65-432b-a863-c39edd6950e4-kube-api-access-jqddf\") pod \"payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7\" (UID: \"6e82685b-9c65-432b-a863-c39edd6950e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:36.111412 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.107982 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpzj\" (UniqueName: \"kubernetes.io/projected/abd590d7-b8b0-4356-9566-c3e62499c87b-kube-api-access-kbpzj\") pod \"payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8\" (UID: \"abd590d7-b8b0-4356-9566-c3e62499c87b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:36.334562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.334474 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:36.388642 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.388616 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:36.391453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.389285 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.391453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.390798 4862 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 30 18:50:36.420285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.409692 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.450291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.445363 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-wwft7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:36.450291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.445419 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.450291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.445516 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-t7sdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:36.450291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.445541 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.450291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.445635 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7jqjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:36.450291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.445658 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.554498 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.554456 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.626597 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.611371 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 30 18:50:36.626597 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.611894 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 18:50:36.626597 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.613543 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:36.633297 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:36.632789 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:37.063352 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.063321 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364a4ab5-d003-44dc-b85c-6d308fe9ed03" path="/var/lib/kubelet/pods/364a4ab5-d003-44dc-b85c-6d308fe9ed03/volumes" Oct 30 18:50:37.095618 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.095595 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7"] Oct 30 18:50:37.097308 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:50:37.097284 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e82685b_9c65_432b_a863_c39edd6950e4.slice/crio-1e6d8d4aec4919af41a6d1d2fed0d01d41da53925c08bc88b9af62227dfd55ad WatchSource:0}: Error finding container 1e6d8d4aec4919af41a6d1d2fed0d01d41da53925c08bc88b9af62227dfd55ad: Status 404 returned error can't find the container with id 1e6d8d4aec4919af41a6d1d2fed0d01d41da53925c08bc88b9af62227dfd55ad Oct 30 18:50:37.166357 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:50:37.166325 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd590d7_b8b0_4356_9566_c3e62499c87b.slice/crio-73953f4c08394be8186de24f3c6f6f7aab4f8fe27d741e983a1bc5fcc00f89e3 WatchSource:0}: Error finding container 73953f4c08394be8186de24f3c6f6f7aab4f8fe27d741e983a1bc5fcc00f89e3: Status 404 returned error can't find the container with id 73953f4c08394be8186de24f3c6f6f7aab4f8fe27d741e983a1bc5fcc00f89e3 Oct 30 18:50:37.166599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.166370 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8"] Oct 30 18:50:37.677716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.677670 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" event={"ID":"abd590d7-b8b0-4356-9566-c3e62499c87b","Type":"ContainerStarted","Data":"16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11"} Oct 30 18:50:37.677934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.677776 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" event={"ID":"abd590d7-b8b0-4356-9566-c3e62499c87b","Type":"ContainerStarted","Data":"73953f4c08394be8186de24f3c6f6f7aab4f8fe27d741e983a1bc5fcc00f89e3"} Oct 30 18:50:37.677934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.677906 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:37.679569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.679542 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" event={"ID":"6e82685b-9c65-432b-a863-c39edd6950e4","Type":"ContainerStarted","Data":"a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1"} Oct 30 18:50:37.679712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.679582 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" event={"ID":"6e82685b-9c65-432b-a863-c39edd6950e4","Type":"ContainerStarted","Data":"1e6d8d4aec4919af41a6d1d2fed0d01d41da53925c08bc88b9af62227dfd55ad"} Oct 30 18:50:37.679712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.679612 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:37.719300 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.719208 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" podStartSLOduration=3.71919088 podStartE2EDuration="3.71919088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:37.715919477 +0000 UTC m=+3136.764964390" watchObservedRunningTime="2026-10-30 18:50:37.71919088 +0000 UTC m=+3136.768235795" Oct 30 18:50:37.789213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.788359 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" podStartSLOduration=2.788339062 podStartE2EDuration="2.788339062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:37.781613177 +0000 UTC m=+3136.830658092" watchObservedRunningTime="2026-10-30 18:50:37.788339062 +0000 UTC m=+3136.837383976" Oct 30 18:50:37.810095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.810048 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb"] Oct 30 18:50:37.814820 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.814799 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:37.830719 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.830692 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:50:37.830876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.830814 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:50:37.834729 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.834701 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb"] Oct 30 18:50:37.958291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.958197 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f21fa4c-n8vrb\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:37.958445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.958300 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f21fa4c-n8vrb\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:37.958445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:37.958348 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8q95\" (UniqueName: \"kubernetes.io/projected/434920f1-3ab4-4902-8b58-a65c114c968c-kube-api-access-w8q95\") pod \"maas-api-revoke-sub-7f21fa4c-n8vrb\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:38.059694 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:38.059665 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f21fa4c-n8vrb\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:38.059861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:38.059724 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f21fa4c-n8vrb\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:38.059861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:38.059757 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8q95\" (UniqueName: \"kubernetes.io/projected/434920f1-3ab4-4902-8b58-a65c114c968c-kube-api-access-w8q95\") pod \"maas-api-revoke-sub-7f21fa4c-n8vrb\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:38.060334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:38.060307 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f21fa4c-n8vrb\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:38.062208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:38.062190 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f21fa4c-n8vrb\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:38.079056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:38.079026 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8q95\" (UniqueName: \"kubernetes.io/projected/434920f1-3ab4-4902-8b58-a65c114c968c-kube-api-access-w8q95\") pod \"maas-api-revoke-sub-7f21fa4c-n8vrb\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:38.134370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:38.134348 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:38.447116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:38.447079 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb"] Oct 30 18:50:38.454457 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:50:38.454426 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434920f1_3ab4_4902_8b58_a65c114c968c.slice/crio-5853a10ebe72388951dffc24d6a4cf4a41aada9358407bdbcdf9b90fb09b9489 WatchSource:0}: Error finding container 5853a10ebe72388951dffc24d6a4cf4a41aada9358407bdbcdf9b90fb09b9489: Status 404 returned error can't find the container with id 5853a10ebe72388951dffc24d6a4cf4a41aada9358407bdbcdf9b90fb09b9489 Oct 30 18:50:38.689688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:38.689649 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" event={"ID":"434920f1-3ab4-4902-8b58-a65c114c968c","Type":"ContainerStarted","Data":"5853a10ebe72388951dffc24d6a4cf4a41aada9358407bdbcdf9b90fb09b9489"} Oct 30 18:50:39.699325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:39.699290 4862 generic.go:356] "Generic (PLEG): container finished" podID="434920f1-3ab4-4902-8b58-a65c114c968c" containerID="a07a8a1725128cd0e2b2c5bee4838eabc2ecc0fd2b00dd34747f8756f5b08eca" exitCode=0 Oct 30 18:50:39.699744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:39.699334 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" event={"ID":"434920f1-3ab4-4902-8b58-a65c114c968c","Type":"ContainerDied","Data":"a07a8a1725128cd0e2b2c5bee4838eabc2ecc0fd2b00dd34747f8756f5b08eca"} Oct 30 18:50:40.874292 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:40.874237 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:40.999783 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:40.999706 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-openshift-ca\") pod \"434920f1-3ab4-4902-8b58-a65c114c968c\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " Oct 30 18:50:40.999783 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:40.999777 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-xks-ca\") pod \"434920f1-3ab4-4902-8b58-a65c114c968c\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " Oct 30 18:50:41.000040 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.000021 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434920f1-3ab4-4902-8b58-a65c114c968c-kube-api-access-w8q95\" (UniqueName: \"kubernetes.io/projected/434920f1-3ab4-4902-8b58-a65c114c968c-kube-api-access-w8q95\") pod \"434920f1-3ab4-4902-8b58-a65c114c968c\" (UID: \"434920f1-3ab4-4902-8b58-a65c114c968c\") " Oct 30 18:50:41.000081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.000050 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-openshift-ca" pod "434920f1-3ab4-4902-8b58-a65c114c968c" (UID: "434920f1-3ab4-4902-8b58-a65c114c968c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:41.000652 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.000624 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:50:41.002018 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.001994 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-xks-ca" pod "434920f1-3ab4-4902-8b58-a65c114c968c" (UID: "434920f1-3ab4-4902-8b58-a65c114c968c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:41.002144 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.002127 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434920f1-3ab4-4902-8b58-a65c114c968c-kube-api-access-w8q95" pod "434920f1-3ab4-4902-8b58-a65c114c968c" (UID: "434920f1-3ab4-4902-8b58-a65c114c968c"). InnerVolumeSpecName "kube-api-access-w8q95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:41.101875 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.101850 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434920f1-3ab4-4902-8b58-a65c114c968c-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:50:41.101875 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.101874 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8q95\" (UniqueName: \"kubernetes.io/projected/434920f1-3ab4-4902-8b58-a65c114c968c-kube-api-access-w8q95\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:50:41.715442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.715412 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" Oct 30 18:50:41.715622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.715448 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb" event={"ID":"434920f1-3ab4-4902-8b58-a65c114c968c","Type":"ContainerDied","Data":"5853a10ebe72388951dffc24d6a4cf4a41aada9358407bdbcdf9b90fb09b9489"} Oct 30 18:50:41.715622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:41.715497 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5853a10ebe72388951dffc24d6a4cf4a41aada9358407bdbcdf9b90fb09b9489" Oct 30 18:50:48.030144 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:48.030103 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw"] Oct 30 18:50:48.042438 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:48.042413 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cae8e5e-dpksw"] Oct 30 18:50:48.693134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:48.693107 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 18:50:48.693316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:48.693172 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 18:50:49.064359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:49.064327 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b0957a-bbbd-44fa-b58c-00ec8d15cd10" path="/var/lib/kubelet/pods/76b0957a-bbbd-44fa-b58c-00ec8d15cd10/volumes" Oct 30 18:50:49.870407 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:50:49.870373 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:50:50.811137 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:50.811104 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr"] Oct 30 18:50:53.260081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:53.260044 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:50:54.036951 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:54.036919 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6"] Oct 30 18:50:54.048849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:54.048828 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7982b6-ppbd6"] Oct 30 18:50:55.063158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:55.063117 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3" path="/var/lib/kubelet/pods/4dac5ae7-a4ab-4ce1-bcce-d20bcdab0bf3/volumes" Oct 30 18:50:55.812574 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:55.812531 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" podUID="9caef399-ed47-4777-9215-df9b20da7242" containerName="payload-processing" containerID="cri-o://c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28" gracePeriod=25 Oct 30 18:50:56.159773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.159746 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:50:56.260972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.260939 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-provider-credentials\") pod \"9caef399-ed47-4777-9215-df9b20da7242\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " Oct 30 18:50:56.261182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.261103 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-plugins-config-volume\") pod \"9caef399-ed47-4777-9215-df9b20da7242\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " Oct 30 18:50:56.261182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.261142 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-kube-api-access-p6njj\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-kube-api-access-p6njj\") pod \"9caef399-ed47-4777-9215-df9b20da7242\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " Oct 30 18:50:56.261422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.261401 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-routing-overlay-volume\") pod \"9caef399-ed47-4777-9215-df9b20da7242\" (UID: \"9caef399-ed47-4777-9215-df9b20da7242\") " Oct 30 18:50:56.261638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.261559 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-plugins-config-volume" pod "9caef399-ed47-4777-9215-df9b20da7242" (UID: "9caef399-ed47-4777-9215-df9b20da7242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:56.261881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.261767 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-routing-overlay-volume" pod "9caef399-ed47-4777-9215-df9b20da7242" (UID: "9caef399-ed47-4777-9215-df9b20da7242"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:56.262123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.262106 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:50:56.262208 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.262129 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9caef399-ed47-4777-9215-df9b20da7242-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:50:56.263626 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.263606 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-kube-api-access-p6njj" pod "9caef399-ed47-4777-9215-df9b20da7242" (UID: "9caef399-ed47-4777-9215-df9b20da7242"). InnerVolumeSpecName "kube-api-access-p6njj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:56.263917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.263899 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-provider-credentials" pod "9caef399-ed47-4777-9215-df9b20da7242" (UID: "9caef399-ed47-4777-9215-df9b20da7242"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:56.364010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.363916 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6njj\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-kube-api-access-p6njj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:50:56.364010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.363946 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9caef399-ed47-4777-9215-df9b20da7242-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:50:56.862539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.862505 4862 generic.go:356] "Generic (PLEG): container finished" podID="9caef399-ed47-4777-9215-df9b20da7242" containerID="c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28" exitCode=0 Oct 30 18:50:56.862740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.862558 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" Oct 30 18:50:56.862740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.862593 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" event={"ID":"9caef399-ed47-4777-9215-df9b20da7242","Type":"ContainerDied","Data":"c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28"} Oct 30 18:50:56.862740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.862652 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr" event={"ID":"9caef399-ed47-4777-9215-df9b20da7242","Type":"ContainerDied","Data":"32aa2f4d39c8cd9dd0afa4f7441b2c39ca10f8be20d6b2125fc9f95a34ca3779"} Oct 30 18:50:56.862740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.862681 4862 scope.go:122] "RemoveContainer" containerID="c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28" Oct 30 18:50:56.902001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.901981 4862 scope.go:122] "RemoveContainer" containerID="c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28" Oct 30 18:50:56.902345 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:50:56.902327 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28\": container with ID starting with c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28 not found: ID does not exist" containerID="c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28" Oct 30 18:50:56.902428 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.902353 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28"} err="failed to get container status \"c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28\": rpc error: code = NotFound desc = could not find container \"c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28\": container with ID starting with c2a59fa7c30d600330c9953f3328f8b5de58d0a4708374cb0c6fa0979c324d28 not found: ID does not exist" Oct 30 18:50:56.921720 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.921696 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr"] Oct 30 18:50:56.955928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:56.955898 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6656cd76c4-lf7dr"] Oct 30 18:50:57.063803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:50:57.063765 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9caef399-ed47-4777-9215-df9b20da7242" path="/var/lib/kubelet/pods/9caef399-ed47-4777-9215-df9b20da7242/volumes" Oct 30 18:51:01.046947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:01.044622 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2"] Oct 30 18:51:01.066038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:01.065996 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2"] Oct 30 18:51:01.073407 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:51:01.073371 4862 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-2819b79e-rbdb2\" is forbidden: User \"system:node:ip-10-0-128-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-246.ec2.internal' and this object" podUID="df54a9fa-e594-47a5-b78b-bfa91fe6dfc6" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2819b79e-rbdb2" Oct 30 18:51:03.100298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:03.099216 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df54a9fa-e594-47a5-b78b-bfa91fe6dfc6" path="/var/lib/kubelet/pods/df54a9fa-e594-47a5-b78b-bfa91fe6dfc6/volumes" Oct 30 18:51:04.930397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:04.930253 4862 generic.go:356] "Generic (PLEG): container finished" podID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerID="d1a9c0e4430e76c95d953e88a5d8cbcfcfa6c48dd84c1555262c2f1ed8c61b10" exitCode=0 Oct 30 18:51:04.930868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:04.930631 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" event={"ID":"4e97e48f-6b9d-4832-9876-18c1a03d431a","Type":"ContainerDied","Data":"d1a9c0e4430e76c95d953e88a5d8cbcfcfa6c48dd84c1555262c2f1ed8c61b10"} Oct 30 18:51:04.931014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:04.930998 4862 scope.go:122] "RemoveContainer" containerID="d1a9c0e4430e76c95d953e88a5d8cbcfcfa6c48dd84c1555262c2f1ed8c61b10" Oct 30 18:51:04.932918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:04.932888 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:51:04.933053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:04.932968 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:51:04.933053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:04.933004 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:51:05.963968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:05.963936 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" event={"ID":"4e97e48f-6b9d-4832-9876-18c1a03d431a","Type":"ContainerStarted","Data":"ae43945ff615dd128905956d20fbefea21a5fb7359b9ab741315671641875338"} Oct 30 18:51:06.062504 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.062473 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k"] Oct 30 18:51:06.063133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.063119 4862 cpu_manager.go:414] "containerMap: removing container" podUID="434920f1-3ab4-4902-8b58-a65c114c968c" containerName="revoke-keys" Oct 30 18:51:06.063192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.063136 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434920f1-3ab4-4902-8b58-a65c114c968c" containerName="revoke-keys" Oct 30 18:51:06.063192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.063148 4862 cpu_manager.go:414] "containerMap: removing container" podUID="9caef399-ed47-4777-9215-df9b20da7242" containerName="payload-processing" Oct 30 18:51:06.063192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.063153 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9caef399-ed47-4777-9215-df9b20da7242" containerName="payload-processing" Oct 30 18:51:06.063321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.063248 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="434920f1-3ab4-4902-8b58-a65c114c968c" containerName="revoke-keys" Oct 30 18:51:06.063321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.063282 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="9caef399-ed47-4777-9215-df9b20da7242" containerName="payload-processing" Oct 30 18:51:06.067125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.067109 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:06.081577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.081553 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8862aa\"" Oct 30 18:51:06.081705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.081673 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8862aa-dockercfg-wjdwn\"" Oct 30 18:51:06.084053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.084028 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k"] Oct 30 18:51:06.345959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.334893 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k\" (UID: \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:06.346575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.346543 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gwg7\" (UniqueName: \"kubernetes.io/projected/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-kube-api-access-8gwg7\") pod \"payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k\" (UID: \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:06.357159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.357130 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct"] Oct 30 18:51:06.371540 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.371514 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct"] Oct 30 18:51:06.371851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.371801 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:06.452286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.449946 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k\" (UID: \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:06.452286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.450188 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gwg7\" (UniqueName: \"kubernetes.io/projected/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-kube-api-access-8gwg7\") pod \"payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k\" (UID: \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:06.455422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.455391 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k\" (UID: \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:06.475166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.474108 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gwg7\" (UniqueName: \"kubernetes.io/projected/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-kube-api-access-8gwg7\") pod \"payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k\" (UID: \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:06.553287 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.552328 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8m8\" (UniqueName: \"kubernetes.io/projected/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-kube-api-access-8h8m8\") pod \"payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct\" (UID: \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:06.553499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.553378 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct\" (UID: \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:06.655228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.655128 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8m8\" (UniqueName: \"kubernetes.io/projected/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-kube-api-access-8h8m8\") pod \"payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct\" (UID: \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:06.655228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.655179 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct\" (UID: \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:06.655964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.655938 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct\" (UID: \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:06.674194 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.674164 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8m8\" (UniqueName: \"kubernetes.io/projected/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-kube-api-access-8h8m8\") pod \"payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct\" (UID: \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:06.683047 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.683018 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:06.692472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.692455 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:06.919509 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.919432 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:51:06.919715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.919695 4862 patch_prober.go:29] interesting pod/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:51:06.919776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:06.919742 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 18:51:07.105780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.102393 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf"] Oct 30 18:51:07.105780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.102529 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97dec1d2-h6gdf"] Oct 30 18:51:07.108556 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:51:07.108529 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcd772e_bd3a_45f6_a8a0_b1bfb97d9b61.slice/crio-dc0d72652e7a21eadd4cb8987f9774446894bbe0c57763dbb2cf2f3efb47feb8 WatchSource:0}: Error finding container dc0d72652e7a21eadd4cb8987f9774446894bbe0c57763dbb2cf2f3efb47feb8: Status 404 returned error can't find the container with id dc0d72652e7a21eadd4cb8987f9774446894bbe0c57763dbb2cf2f3efb47feb8 Oct 30 18:51:07.116547 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.116525 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a794af-f0fa-42ce-886c-4eb4be691cda" path="/var/lib/kubelet/pods/35a794af-f0fa-42ce-886c-4eb4be691cda/volumes" Oct 30 18:51:07.116894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.116879 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct"] Oct 30 18:51:07.116972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.116918 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k"] Oct 30 18:51:07.919457 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.919422 4862 patch_prober.go:29] interesting pod/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:51:07.919638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.919474 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 18:51:07.979757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.979719 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" event={"ID":"a0c69a88-80c5-4d22-a1b3-172e2f6c171d","Type":"ContainerStarted","Data":"66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb"} Oct 30 18:51:07.979957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.979814 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" event={"ID":"a0c69a88-80c5-4d22-a1b3-172e2f6c171d","Type":"ContainerStarted","Data":"b110b5c9570e8886d81ea96f74da905f06e4ee5b90ca80dfd72495e18e863bf3"} Oct 30 18:51:07.980032 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.979978 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:07.981427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.981398 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" event={"ID":"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61","Type":"ContainerStarted","Data":"65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775"} Oct 30 18:51:07.981591 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.981457 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:07.981591 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:07.981483 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" event={"ID":"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61","Type":"ContainerStarted","Data":"dc0d72652e7a21eadd4cb8987f9774446894bbe0c57763dbb2cf2f3efb47feb8"} Oct 30 18:51:08.022504 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:08.022464 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" podStartSLOduration=2.022451683 podStartE2EDuration="2.022451683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:08.019067619 +0000 UTC m=+3167.068112532" watchObservedRunningTime="2026-10-30 18:51:08.022451683 +0000 UTC m=+3167.071496595" Oct 30 18:51:08.074318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:08.074273 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" podStartSLOduration=2.07424569 podStartE2EDuration="2.07424569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:08.072243171 +0000 UTC m=+3167.121288085" watchObservedRunningTime="2026-10-30 18:51:08.07424569 +0000 UTC m=+3167.123290603" Oct 30 18:51:08.919651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:08.919620 4862 patch_prober.go:29] interesting pod/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:51:08.920072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:08.919669 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 18:51:09.919202 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:09.919166 4862 patch_prober.go:29] interesting pod/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:51:09.919381 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:09.919240 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 18:51:10.919654 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:10.919622 4862 patch_prober.go:29] interesting pod/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:51:10.920045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:10.919673 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 18:51:11.925612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:11.925584 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:51:11.926098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:11.925987 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:51:11.926587 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:11.926567 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 18:51:18.991464 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:18.991438 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 18:51:18.991959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:18.991503 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 18:51:19.870889 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:51:19.870844 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:51:22.280400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:22.280370 4862 scope.go:122] "RemoveContainer" containerID="431bc637781670ea98ce934bc6eacd4314ca60b7b69f866879093a8220f2ee80" Oct 30 18:51:29.146255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:29.146225 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:51:32.631400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:32.631168 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Oct 30 18:51:36.869719 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.869685 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd"] Oct 30 18:51:36.875783 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.875751 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.879868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.879842 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.880089 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.880069 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.880213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.880192 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.880282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.880216 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25dp8\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-kube-api-access-25dp8\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.901480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.901453 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd"] Oct 30 18:51:36.926205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.926177 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2107d4\"/\"agc-cred-284bd6e1109c\"" Oct 30 18:51:36.981541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.981512 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.981688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.981605 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.981688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.981626 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25dp8\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-kube-api-access-25dp8\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.981688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.981662 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.982520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.982500 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.983106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.983078 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:36.984696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:36.984666 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:37.015087 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.015065 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25dp8\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-kube-api-access-25dp8\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:37.190184 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.190114 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:37.499581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.499540 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd"] Oct 30 18:51:37.504089 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:51:37.504058 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9161c1fc_5591_4b85_97c9_321063263979.slice/crio-37a1516ac078b2114343df4c1680d82ed01418c556f7ce940ae41d8769af42df WatchSource:0}: Error finding container 37a1516ac078b2114343df4c1680d82ed01418c556f7ce940ae41d8769af42df: Status 404 returned error can't find the container with id 37a1516ac078b2114343df4c1680d82ed01418c556f7ce940ae41d8769af42df Oct 30 18:51:37.558912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.558881 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7"] Oct 30 18:51:37.567195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.567162 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.582877 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.582806 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-551d0b-openshift-default-dockercfg-r95tk\"" Oct 30 18:51:37.600221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.600196 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7"] Oct 30 18:51:37.620934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.620210 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.620934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.620348 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.620934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.620383 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkcd9\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-kube-api-access-zkcd9\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.620934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.620418 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.620934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.620633 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.620934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.620675 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.620934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.620785 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.620934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.620814 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.620934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.620843 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722354 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722318 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722361 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722430 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722449 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722466 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722524 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722581 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722607 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkcd9\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-kube-api-access-zkcd9\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722634 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.722841 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.722772 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.723167 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.723145 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.723291 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.723244 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.723363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.723290 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.723417 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.723402 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.725742 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.725718 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.725941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.725925 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.745030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.744819 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkcd9\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-kube-api-access-zkcd9\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.745543 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.745525 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.912288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.900752 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:37.912288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.906721 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr"] Oct 30 18:51:37.925313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.913408 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:37.925313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.917402 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr"] Oct 30 18:51:37.992410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.989636 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:37.992410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.989683 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lg5\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-kube-api-access-f7lg5\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:37.992410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.989815 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:37.992410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.990003 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:37.992410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:37.990950 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-38e0db\"/\"agc-cred-4018ab138bae\"" Oct 30 18:51:38.092335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.092232 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:38.092476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.092356 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lg5\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-kube-api-access-f7lg5\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:38.092552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.092504 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:38.092716 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.092697 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:38.095142 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.094219 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:38.095142 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.094972 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:38.103397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.103374 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:38.131364 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.130672 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lg5\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-kube-api-access-f7lg5\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:38.358529 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.358497 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7"] Oct 30 18:51:38.359326 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:51:38.359288 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8a371c_43aa_4fca_bf65_c22c553cbb8c.slice/crio-81e2b078c7aa6ce8b035e4c1724dc52bd60002dc60f6296766d7032aaeca17af WatchSource:0}: Error finding container 81e2b078c7aa6ce8b035e4c1724dc52bd60002dc60f6296766d7032aaeca17af: Status 404 returned error can't find the container with id 81e2b078c7aa6ce8b035e4c1724dc52bd60002dc60f6296766d7032aaeca17af Oct 30 18:51:38.362622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.362584 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:51:38.362735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.362652 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:51:38.362735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.362688 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:51:38.371657 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.371635 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:38.423134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.420749 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" event={"ID":"6f8a371c-43aa-4fca-bf65-c22c553cbb8c","Type":"ContainerStarted","Data":"81e2b078c7aa6ce8b035e4c1724dc52bd60002dc60f6296766d7032aaeca17af"} Oct 30 18:51:38.423134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.422377 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" event={"ID":"9161c1fc-5591-4b85-97c9-321063263979","Type":"ContainerStarted","Data":"411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115"} Oct 30 18:51:38.423134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.422416 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" event={"ID":"9161c1fc-5591-4b85-97c9-321063263979","Type":"ContainerStarted","Data":"37a1516ac078b2114343df4c1680d82ed01418c556f7ce940ae41d8769af42df"} Oct 30 18:51:38.423134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.423031 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:38.700633 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.700592 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" podStartSLOduration=2.700579499 podStartE2EDuration="2.700579499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:38.468887457 +0000 UTC m=+3197.517932372" watchObservedRunningTime="2026-10-30 18:51:38.700579499 +0000 UTC m=+3197.749624428" Oct 30 18:51:38.702646 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:38.702624 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr"] Oct 30 18:51:38.704402 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:51:38.704379 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2274b699_f5b5_4ef4_9660_1690e8c68381.slice/crio-6bf4589ff21ee41ef84a4ef38763b5b7b55ee6bcdefc67d0131db440e81941e7 WatchSource:0}: Error finding container 6bf4589ff21ee41ef84a4ef38763b5b7b55ee6bcdefc67d0131db440e81941e7: Status 404 returned error can't find the container with id 6bf4589ff21ee41ef84a4ef38763b5b7b55ee6bcdefc67d0131db440e81941e7 Oct 30 18:51:39.431949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:39.431916 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" event={"ID":"2274b699-f5b5-4ef4-9660-1690e8c68381","Type":"ContainerStarted","Data":"1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332"} Oct 30 18:51:39.432465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:39.431977 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" event={"ID":"2274b699-f5b5-4ef4-9660-1690e8c68381","Type":"ContainerStarted","Data":"6bf4589ff21ee41ef84a4ef38763b5b7b55ee6bcdefc67d0131db440e81941e7"} Oct 30 18:51:39.432465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:39.432006 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:39.433701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:39.433671 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" event={"ID":"6f8a371c-43aa-4fca-bf65-c22c553cbb8c","Type":"ContainerStarted","Data":"3e56ebb0bd4644d0096aefd6d2a89031015022e90b0bd86b97f2a17deceb0510"} Oct 30 18:51:39.471877 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:39.471829 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" podStartSLOduration=2.471816714 podStartE2EDuration="2.471816714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:39.471100656 +0000 UTC m=+3198.520145568" watchObservedRunningTime="2026-10-30 18:51:39.471816714 +0000 UTC m=+3198.520861623" Oct 30 18:51:39.512622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:39.512572 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" podStartSLOduration=2.5125607260000002 podStartE2EDuration="2.512560726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:39.511148823 +0000 UTC m=+3198.560193752" watchObservedRunningTime="2026-10-30 18:51:39.512560726 +0000 UTC m=+3198.561605638" Oct 30 18:51:39.906587 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:39.905440 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:39.914058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:39.913570 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:40.468217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:40.468140 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:40.480929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:40.480903 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:51:42.069689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:42.069658 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb"] Oct 30 18:51:42.082394 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:42.082366 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2873db0-bxmvb"] Oct 30 18:51:43.037817 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:43.037778 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr"] Oct 30 18:51:43.048947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:43.048921 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768844fd-cx2wr"] Oct 30 18:51:43.064414 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:43.064387 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d31eb1c-6d5e-43e0-ac39-36a9cc52746a" path="/var/lib/kubelet/pods/7d31eb1c-6d5e-43e0-ac39-36a9cc52746a/volumes" Oct 30 18:51:43.064844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:43.064830 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26d21de-92bd-4102-9acb-425c3bb2e3bf" path="/var/lib/kubelet/pods/b26d21de-92bd-4102-9acb-425c3bb2e3bf/volumes" Oct 30 18:51:45.635964 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.635937 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj"] Oct 30 18:51:45.641717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.641697 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:45.659542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.659514 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj"] Oct 30 18:51:45.680073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.680038 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wfm\" (UniqueName: \"kubernetes.io/projected/8dd4156c-152e-4192-9333-1f687507ee64-kube-api-access-n4wfm\") pod \"payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj\" (UID: \"8dd4156c-152e-4192-9333-1f687507ee64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:45.680407 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.680382 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4156c-152e-4192-9333-1f687507ee64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj\" (UID: \"8dd4156c-152e-4192-9333-1f687507ee64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:45.752285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.749625 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-551d0b-dockercfg-zfkb9\"" Oct 30 18:51:45.752285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.749776 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-551d0b\"" Oct 30 18:51:45.787499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.787463 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wfm\" (UniqueName: \"kubernetes.io/projected/8dd4156c-152e-4192-9333-1f687507ee64-kube-api-access-n4wfm\") pod \"payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj\" (UID: \"8dd4156c-152e-4192-9333-1f687507ee64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:45.787821 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.787786 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4156c-152e-4192-9333-1f687507ee64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj\" (UID: \"8dd4156c-152e-4192-9333-1f687507ee64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:45.789085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.789048 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4156c-152e-4192-9333-1f687507ee64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj\" (UID: \"8dd4156c-152e-4192-9333-1f687507ee64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:45.844396 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.844072 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wfm\" (UniqueName: \"kubernetes.io/projected/8dd4156c-152e-4192-9333-1f687507ee64-kube-api-access-n4wfm\") pod \"payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj\" (UID: \"8dd4156c-152e-4192-9333-1f687507ee64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:45.959679 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:45.959598 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:46.267092 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:46.267063 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj"] Oct 30 18:51:46.267494 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:51:46.267473 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd4156c_152e_4192_9333_1f687507ee64.slice/crio-087b8257e67f403ed0d31667371a07937007017af15f357b7f7f1680ee5dc68d WatchSource:0}: Error finding container 087b8257e67f403ed0d31667371a07937007017af15f357b7f7f1680ee5dc68d: Status 404 returned error can't find the container with id 087b8257e67f403ed0d31667371a07937007017af15f357b7f7f1680ee5dc68d Oct 30 18:51:46.597089 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:46.597053 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" event={"ID":"8dd4156c-152e-4192-9333-1f687507ee64","Type":"ContainerStarted","Data":"59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7"} Oct 30 18:51:46.597299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:46.597154 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" event={"ID":"8dd4156c-152e-4192-9333-1f687507ee64","Type":"ContainerStarted","Data":"087b8257e67f403ed0d31667371a07937007017af15f357b7f7f1680ee5dc68d"} Oct 30 18:51:46.597299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:46.597182 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:46.636869 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:46.636823 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" podStartSLOduration=1.63681052 podStartE2EDuration="1.63681052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:46.635688846 +0000 UTC m=+3205.684733761" watchObservedRunningTime="2026-10-30 18:51:46.63681052 +0000 UTC m=+3205.685855432" Oct 30 18:51:49.437433 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:49.437404 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:51:49.508333 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:49.508301 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9"] Oct 30 18:51:49.871192 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:51:49.871158 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:51:50.464530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:50.464500 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:51:50.535521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:50.535489 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns"] Oct 30 18:51:53.680556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:53.680523 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv"] Oct 30 18:51:54.510062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.510010 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" podUID="a8060716-556f-4b67-9d34-0b3a6a3675ca" containerName="payload-processing" containerID="cri-o://e5d03105e88ca73a5b813401e1f0ffe20ca1a24947f5f3ea3a90193eeddd0ecf" gracePeriod=25 Oct 30 18:51:54.665434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.665403 4862 generic.go:356] "Generic (PLEG): container finished" podID="a8060716-556f-4b67-9d34-0b3a6a3675ca" containerID="e5d03105e88ca73a5b813401e1f0ffe20ca1a24947f5f3ea3a90193eeddd0ecf" exitCode=0 Oct 30 18:51:54.665613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.665481 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" event={"ID":"a8060716-556f-4b67-9d34-0b3a6a3675ca","Type":"ContainerDied","Data":"e5d03105e88ca73a5b813401e1f0ffe20ca1a24947f5f3ea3a90193eeddd0ecf"} Oct 30 18:51:54.806357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.806331 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:51:54.835330 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.835301 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-kube-api-access-68j5v\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-kube-api-access-68j5v\") pod \"a8060716-556f-4b67-9d34-0b3a6a3675ca\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " Oct 30 18:51:54.835512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.835365 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-plugins-config-volume\") pod \"a8060716-556f-4b67-9d34-0b3a6a3675ca\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " Oct 30 18:51:54.835512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.835432 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-routing-overlay-volume\") pod \"a8060716-556f-4b67-9d34-0b3a6a3675ca\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " Oct 30 18:51:54.835512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.835486 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-provider-credentials\") pod \"a8060716-556f-4b67-9d34-0b3a6a3675ca\" (UID: \"a8060716-556f-4b67-9d34-0b3a6a3675ca\") " Oct 30 18:51:54.835825 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.835800 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-plugins-config-volume" pod "a8060716-556f-4b67-9d34-0b3a6a3675ca" (UID: "a8060716-556f-4b67-9d34-0b3a6a3675ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:54.835925 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.835872 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-routing-overlay-volume" pod "a8060716-556f-4b67-9d34-0b3a6a3675ca" (UID: "a8060716-556f-4b67-9d34-0b3a6a3675ca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:54.836150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.836126 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:54.836285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.836155 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a8060716-556f-4b67-9d34-0b3a6a3675ca-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:54.837822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.837794 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-provider-credentials" pod "a8060716-556f-4b67-9d34-0b3a6a3675ca" (UID: "a8060716-556f-4b67-9d34-0b3a6a3675ca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:54.837909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.837823 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-kube-api-access-68j5v" pod "a8060716-556f-4b67-9d34-0b3a6a3675ca" (UID: "a8060716-556f-4b67-9d34-0b3a6a3675ca"). InnerVolumeSpecName "kube-api-access-68j5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:54.937822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.937781 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68j5v\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-kube-api-access-68j5v\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:54.937822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:54.937824 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a8060716-556f-4b67-9d34-0b3a6a3675ca-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:55.543355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.541380 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" podUID="c684c0e8-e10e-434c-a7d8-cd0d38822e73" containerName="payload-processing" containerID="cri-o://592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a" gracePeriod=25 Oct 30 18:51:55.674394 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.674363 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" event={"ID":"a8060716-556f-4b67-9d34-0b3a6a3675ca","Type":"ContainerDied","Data":"b39846b604fa12fa765cd20eab8638c5aba6fd9aa0f4e94a67b2709a32f779c0"} Oct 30 18:51:55.674571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.674404 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9" Oct 30 18:51:55.674571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.674449 4862 scope.go:122] "RemoveContainer" containerID="e5d03105e88ca73a5b813401e1f0ffe20ca1a24947f5f3ea3a90193eeddd0ecf" Oct 30 18:51:55.730779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.730743 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9"] Oct 30 18:51:55.743590 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.743557 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9"] Oct 30 18:51:55.749475 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.749450 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784jndt9"] Oct 30 18:51:55.889254 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.889225 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:51:55.960316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.960248 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-plugins-config-volume\") pod \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " Oct 30 18:51:55.960495 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.960402 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-provider-credentials\") pod \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " Oct 30 18:51:55.960581 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.960563 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-routing-overlay-volume\") pod \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " Oct 30 18:51:55.960698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.960679 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-plugins-config-volume" pod "c684c0e8-e10e-434c-a7d8-cd0d38822e73" (UID: "c684c0e8-e10e-434c-a7d8-cd0d38822e73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:55.960776 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.960741 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-kube-api-access-p54qj\") pod \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\" (UID: \"c684c0e8-e10e-434c-a7d8-cd0d38822e73\") " Oct 30 18:51:55.960980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.960955 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-routing-overlay-volume" pod "c684c0e8-e10e-434c-a7d8-cd0d38822e73" (UID: "c684c0e8-e10e-434c-a7d8-cd0d38822e73"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:55.961316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.961292 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:55.961316 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.961315 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c684c0e8-e10e-434c-a7d8-cd0d38822e73-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:55.963041 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.963017 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-provider-credentials" pod "c684c0e8-e10e-434c-a7d8-cd0d38822e73" (UID: "c684c0e8-e10e-434c-a7d8-cd0d38822e73"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:55.963471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:55.963445 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-kube-api-access-p54qj" pod "c684c0e8-e10e-434c-a7d8-cd0d38822e73" (UID: "c684c0e8-e10e-434c-a7d8-cd0d38822e73"). InnerVolumeSpecName "kube-api-access-p54qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:56.071452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.066522 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p54qj\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-kube-api-access-p54qj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:56.071452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.066563 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c684c0e8-e10e-434c-a7d8-cd0d38822e73-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:56.686858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.686826 4862 generic.go:356] "Generic (PLEG): container finished" podID="c684c0e8-e10e-434c-a7d8-cd0d38822e73" containerID="592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a" exitCode=0 Oct 30 18:51:56.687045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.686885 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" Oct 30 18:51:56.687045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.686910 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" event={"ID":"c684c0e8-e10e-434c-a7d8-cd0d38822e73","Type":"ContainerDied","Data":"592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a"} Oct 30 18:51:56.687045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.686969 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns" event={"ID":"c684c0e8-e10e-434c-a7d8-cd0d38822e73","Type":"ContainerDied","Data":"5dbd517725a1d6e167c17247a42de9865b7bdb3dea17bea1aaebb79b7975f3d5"} Oct 30 18:51:56.687045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.687000 4862 scope.go:122] "RemoveContainer" containerID="592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a" Oct 30 18:51:56.698812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.698792 4862 scope.go:122] "RemoveContainer" containerID="592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a" Oct 30 18:51:56.699074 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:51:56.699053 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a\": container with ID starting with 592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a not found: ID does not exist" containerID="592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a" Oct 30 18:51:56.699125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.699085 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a"} err="failed to get container status \"592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a\": rpc error: code = NotFound desc = could not find container \"592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a\": container with ID starting with 592c84a26ad8230f35d35e36c358bd30096a54980b897e02af991f1e2e72932a not found: ID does not exist" Oct 30 18:51:56.739428 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.739404 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns"] Oct 30 18:51:56.751305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:56.751287 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d7sqlns"] Oct 30 18:51:57.063671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:57.063642 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8060716-556f-4b67-9d34-0b3a6a3675ca" path="/var/lib/kubelet/pods/a8060716-556f-4b67-9d34-0b3a6a3675ca/volumes" Oct 30 18:51:57.064090 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:57.064076 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c684c0e8-e10e-434c-a7d8-cd0d38822e73" path="/var/lib/kubelet/pods/c684c0e8-e10e-434c-a7d8-cd0d38822e73/volumes" Oct 30 18:51:57.608610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:57.608584 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 18:51:58.681301 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:58.681238 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" podUID="65fe30b3-06b6-4fd5-a63d-76d131985e73" containerName="payload-processing" containerID="cri-o://2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8" gracePeriod=25 Oct 30 18:51:58.966560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:58.966534 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:51:59.009855 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.009828 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-routing-overlay-volume\") pod \"65fe30b3-06b6-4fd5-a63d-76d131985e73\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " Oct 30 18:51:59.009991 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.009924 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-provider-credentials\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-provider-credentials\") pod \"65fe30b3-06b6-4fd5-a63d-76d131985e73\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " Oct 30 18:51:59.010058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.010038 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-kube-api-access-2pt9d\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-kube-api-access-2pt9d\") pod \"65fe30b3-06b6-4fd5-a63d-76d131985e73\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " Oct 30 18:51:59.010232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.010207 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-routing-overlay-volume" pod "65fe30b3-06b6-4fd5-a63d-76d131985e73" (UID: "65fe30b3-06b6-4fd5-a63d-76d131985e73"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:59.010398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.010255 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-plugins-config-volume\") pod \"65fe30b3-06b6-4fd5-a63d-76d131985e73\" (UID: \"65fe30b3-06b6-4fd5-a63d-76d131985e73\") " Oct 30 18:51:59.010690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.010668 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-plugins-config-volume" pod "65fe30b3-06b6-4fd5-a63d-76d131985e73" (UID: "65fe30b3-06b6-4fd5-a63d-76d131985e73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:59.010870 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.010853 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:59.010944 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.010872 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/65fe30b3-06b6-4fd5-a63d-76d131985e73-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:59.012807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.012787 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-kube-api-access-2pt9d" pod "65fe30b3-06b6-4fd5-a63d-76d131985e73" (UID: "65fe30b3-06b6-4fd5-a63d-76d131985e73"). InnerVolumeSpecName "kube-api-access-2pt9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:59.012905 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.012820 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-provider-credentials" pod "65fe30b3-06b6-4fd5-a63d-76d131985e73" (UID: "65fe30b3-06b6-4fd5-a63d-76d131985e73"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:59.112240 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.112210 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pt9d\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-kube-api-access-2pt9d\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:59.112401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.112248 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/65fe30b3-06b6-4fd5-a63d-76d131985e73-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:51:59.720784 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.720749 4862 generic.go:356] "Generic (PLEG): container finished" podID="65fe30b3-06b6-4fd5-a63d-76d131985e73" containerID="2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8" exitCode=0 Oct 30 18:51:59.721337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.720893 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" event={"ID":"65fe30b3-06b6-4fd5-a63d-76d131985e73","Type":"ContainerDied","Data":"2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8"} Oct 30 18:51:59.721337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.720960 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" event={"ID":"65fe30b3-06b6-4fd5-a63d-76d131985e73","Type":"ContainerDied","Data":"8740a25740fb7e4f59074ecfe2eae7b13f5e3156f41a9ab897f66efa4b32772b"} Oct 30 18:51:59.721337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.720985 4862 scope.go:122] "RemoveContainer" containerID="2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8" Oct 30 18:51:59.721337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.721107 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv" Oct 30 18:51:59.733910 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.733885 4862 scope.go:122] "RemoveContainer" containerID="2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8" Oct 30 18:51:59.734401 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:51:59.734373 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8\": container with ID starting with 2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8 not found: ID does not exist" containerID="2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8" Oct 30 18:51:59.734514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.734409 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8"} err="failed to get container status \"2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8\": rpc error: code = NotFound desc = could not find container \"2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8\": container with ID starting with 2b2362f3d57682432e6e5096cc388bbcf18672d3c43b772ca894d305215ad7f8 not found: ID does not exist" Oct 30 18:51:59.773390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.773362 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv"] Oct 30 18:51:59.785204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:51:59.785169 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-861bb3/payload-processing-external-model-e2e-extmod-a-861bb3-d475ltvhv"] Oct 30 18:52:00.744425 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:00.744384 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" podUID="c727e427-533b-4fbf-933c-701ed44b9e9e" containerName="payload-processing" containerID="cri-o://e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8" gracePeriod=25 Oct 30 18:52:01.082295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.082243 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65fe30b3-06b6-4fd5-a63d-76d131985e73" path="/var/lib/kubelet/pods/65fe30b3-06b6-4fd5-a63d-76d131985e73/volumes" Oct 30 18:52:01.114029 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.114005 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:52:01.255911 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.255878 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-kube-api-access-fh75t\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-kube-api-access-fh75t\") pod \"c727e427-533b-4fbf-933c-701ed44b9e9e\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " Oct 30 18:52:01.256088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.255995 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-provider-credentials\") pod \"c727e427-533b-4fbf-933c-701ed44b9e9e\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " Oct 30 18:52:01.256160 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.256144 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-plugins-config-volume\") pod \"c727e427-533b-4fbf-933c-701ed44b9e9e\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " Oct 30 18:52:01.256324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.256256 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-routing-overlay-volume\") pod \"c727e427-533b-4fbf-933c-701ed44b9e9e\" (UID: \"c727e427-533b-4fbf-933c-701ed44b9e9e\") " Oct 30 18:52:01.256639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.256609 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-plugins-config-volume" pod "c727e427-533b-4fbf-933c-701ed44b9e9e" (UID: "c727e427-533b-4fbf-933c-701ed44b9e9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:01.256759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.256652 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-routing-overlay-volume" pod "c727e427-533b-4fbf-933c-701ed44b9e9e" (UID: "c727e427-533b-4fbf-933c-701ed44b9e9e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:01.256829 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.256815 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:52:01.256870 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.256833 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c727e427-533b-4fbf-933c-701ed44b9e9e-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:52:01.258256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.258233 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-provider-credentials" pod "c727e427-533b-4fbf-933c-701ed44b9e9e" (UID: "c727e427-533b-4fbf-933c-701ed44b9e9e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:01.258353 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.258292 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-kube-api-access-fh75t" pod "c727e427-533b-4fbf-933c-701ed44b9e9e" (UID: "c727e427-533b-4fbf-933c-701ed44b9e9e"). InnerVolumeSpecName "kube-api-access-fh75t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:01.358062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.357990 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:52:01.358062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.358018 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh75t\" (UniqueName: \"kubernetes.io/projected/c727e427-533b-4fbf-933c-701ed44b9e9e-kube-api-access-fh75t\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:52:01.742320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.742208 4862 generic.go:356] "Generic (PLEG): container finished" podID="c727e427-533b-4fbf-933c-701ed44b9e9e" containerID="e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8" exitCode=0 Oct 30 18:52:01.742320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.742273 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" Oct 30 18:52:01.742320 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.742292 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" event={"ID":"c727e427-533b-4fbf-933c-701ed44b9e9e","Type":"ContainerDied","Data":"e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8"} Oct 30 18:52:01.742601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.742341 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9" event={"ID":"c727e427-533b-4fbf-933c-701ed44b9e9e","Type":"ContainerDied","Data":"0fa583805369d5c27df469264634d66a884c569f740ad4e67d495a5b1bfe2ae3"} Oct 30 18:52:01.742601 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.742363 4862 scope.go:122] "RemoveContainer" containerID="e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8" Oct 30 18:52:01.754536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.754386 4862 scope.go:122] "RemoveContainer" containerID="e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8" Oct 30 18:52:01.754760 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:52:01.754644 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8\": container with ID starting with e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8 not found: ID does not exist" containerID="e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8" Oct 30 18:52:01.754760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.754665 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8"} err="failed to get container status \"e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8\": rpc error: code = NotFound desc = could not find container \"e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8\": container with ID starting with e820f549d306bb4e8330ff29e334eec3c64cccec75b1f16e88688a837c8b22b8 not found: ID does not exist" Oct 30 18:52:01.795069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.795037 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9"] Oct 30 18:52:01.806939 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:01.806908 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5t7nd9"] Oct 30 18:52:03.063712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:03.063678 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c727e427-533b-4fbf-933c-701ed44b9e9e" path="/var/lib/kubelet/pods/c727e427-533b-4fbf-933c-701ed44b9e9e/volumes" Oct 30 18:52:14.030213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:14.030134 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l"] Oct 30 18:52:14.042283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:14.042231 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2e261ba-tmj7l"] Oct 30 18:52:15.063780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:15.063749 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0e9d63-ce79-4090-adf1-454819cc179c" path="/var/lib/kubelet/pods/7e0e9d63-ce79-4090-adf1-454819cc179c/volumes" Oct 30 18:52:19.871839 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:52:19.871802 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:52:22.314302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:22.314254 4862 scope.go:122] "RemoveContainer" containerID="94c5f5fbaa5f8d8fec97acd3be6ce30435296e9a95d07f77e01c663a75bda4f0" Oct 30 18:52:22.324794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:22.324775 4862 scope.go:122] "RemoveContainer" containerID="99f04fbbe2cf0d18a2dbbbd36ea94fb5a2deab7bd92f579e63841bc7be3e02fd" Oct 30 18:52:22.333746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:22.333715 4862 scope.go:122] "RemoveContainer" containerID="b5b21e1a777d4bb0c4743e59d7208a13978df152e80d4697317edde381fafbc6" Oct 30 18:52:22.342492 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:22.342476 4862 scope.go:122] "RemoveContainer" containerID="71ff4ea914340138f899d2ee08644d0764e8c13568c18ba2f60bc7f7a39cc54f" Oct 30 18:52:22.353981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:22.353950 4862 scope.go:122] "RemoveContainer" containerID="8accace7a2770a93927b3de3256dc5fbc751d206e75e8aa65a1ec4aba8a3668c" Oct 30 18:52:22.362971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:22.362952 4862 scope.go:122] "RemoveContainer" containerID="00a6f2b7fa737bba611ee660eff4849643a060cfaf50b1e8e593ff06c1d9999e" Oct 30 18:52:26.039033 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:26.038999 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz"] Oct 30 18:52:26.042478 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:26.042451 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c13625d5-ppbwz"] Oct 30 18:52:27.069464 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:27.069430 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b8a80b-3296-47ff-9301-3a9ad215c6d2" path="/var/lib/kubelet/pods/26b8a80b-3296-47ff-9301-3a9ad215c6d2/volumes" Oct 30 18:52:28.821437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.821402 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk"] Oct 30 18:52:28.822488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822463 4862 cpu_manager.go:414] "containerMap: removing container" podUID="65fe30b3-06b6-4fd5-a63d-76d131985e73" containerName="payload-processing" Oct 30 18:52:28.822488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822489 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65fe30b3-06b6-4fd5-a63d-76d131985e73" containerName="payload-processing" Oct 30 18:52:28.822689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822507 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c727e427-533b-4fbf-933c-701ed44b9e9e" containerName="payload-processing" Oct 30 18:52:28.822689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822515 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c727e427-533b-4fbf-933c-701ed44b9e9e" containerName="payload-processing" Oct 30 18:52:28.822689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822545 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a8060716-556f-4b67-9d34-0b3a6a3675ca" containerName="payload-processing" Oct 30 18:52:28.822689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822553 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8060716-556f-4b67-9d34-0b3a6a3675ca" containerName="payload-processing" Oct 30 18:52:28.822689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822585 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c684c0e8-e10e-434c-a7d8-cd0d38822e73" containerName="payload-processing" Oct 30 18:52:28.822689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822593 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c684c0e8-e10e-434c-a7d8-cd0d38822e73" containerName="payload-processing" Oct 30 18:52:28.823010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822844 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="65fe30b3-06b6-4fd5-a63d-76d131985e73" containerName="payload-processing" Oct 30 18:52:28.823010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822862 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c727e427-533b-4fbf-933c-701ed44b9e9e" containerName="payload-processing" Oct 30 18:52:28.823010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822879 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8060716-556f-4b67-9d34-0b3a6a3675ca" containerName="payload-processing" Oct 30 18:52:28.823010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.822891 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c684c0e8-e10e-434c-a7d8-cd0d38822e73" containerName="payload-processing" Oct 30 18:52:28.830114 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.830077 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:28.846334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.846310 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk"] Oct 30 18:52:28.853070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.853046 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0cc6b1\"/\"default-dockercfg-pv56l\"" Oct 30 18:52:28.853214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.853127 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0cc6b1\"/\"test-model-0cc6b1-kserve-self-signed-certs\"" Oct 30 18:52:28.915802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.915773 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0cc6b1\"/\"kube-root-ca.crt\"" Oct 30 18:52:28.915802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.915773 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0cc6b1\"/\"openshift-service-ca.crt\"" Oct 30 18:52:28.983812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.983768 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-dshm\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:28.984047 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.983925 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tls-certs\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:28.984047 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.984010 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tmp-dir\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:28.984182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.984050 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-model-cache\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:28.984182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.984124 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-home\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:28.984357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:28.984232 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbr7\" (UniqueName: \"kubernetes.io/projected/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-kube-api-access-zlbr7\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.086343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.085712 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tls-certs\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.086343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.085767 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tmp-dir\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.086343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.085794 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-model-cache\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.086343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.085831 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-home\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.086343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.085919 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbr7\" (UniqueName: \"kubernetes.io/projected/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-kube-api-access-zlbr7\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.086343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.086163 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-dshm\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.087481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.087454 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tmp-dir\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.091203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.091178 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-model-cache\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.096889 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.096870 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-home\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.122285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.104614 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tls-certs\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.123220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.105633 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-dshm\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.132111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.132048 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbr7\" (UniqueName: \"kubernetes.io/projected/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-kube-api-access-zlbr7\") pod \"test-model-0cc6b1-kserve-759bb69b69-j6hhk\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.152286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.151577 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:29.510033 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:52:29.510005 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbe93ef3_4e4b_45e5_b3ea_12ae43e224c9.slice/crio-750dfe655378a74429f9df29adeabae22c1c8776dcab08173ac5b3dc4c7d7b72 WatchSource:0}: Error finding container 750dfe655378a74429f9df29adeabae22c1c8776dcab08173ac5b3dc4c7d7b72: Status 404 returned error can't find the container with id 750dfe655378a74429f9df29adeabae22c1c8776dcab08173ac5b3dc4c7d7b72 Oct 30 18:52:29.511695 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.511662 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk"] Oct 30 18:52:29.959093 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.959056 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" event={"ID":"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9","Type":"ContainerStarted","Data":"4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627"} Oct 30 18:52:29.959569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:29.959106 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" event={"ID":"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9","Type":"ContainerStarted","Data":"750dfe655378a74429f9df29adeabae22c1c8776dcab08173ac5b3dc4c7d7b72"} Oct 30 18:52:30.000155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:30.000109 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" podStartSLOduration=2.000095779 podStartE2EDuration="2.000095779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:29.996945369 +0000 UTC m=+3249.045990275" watchObservedRunningTime="2026-10-30 18:52:30.000095779 +0000 UTC m=+3249.049140703" Oct 30 18:52:30.155548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:30.153363 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:33.720448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:33.720409 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:52:39.154156 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:39.154124 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:39.167448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:39.167425 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:40.084417 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:40.084387 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:52:44.031998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.031968 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272"] Oct 30 18:52:44.044009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.043977 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44d54f07-7z272"] Oct 30 18:52:44.450794 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.450763 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg"] Oct 30 18:52:44.455779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.455756 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.486920 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.486877 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8862aa-openshift-default-dockercfg-5svft\"" Oct 30 18:52:44.493370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.493341 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg"] Oct 30 18:52:44.538680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.538648 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.538680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.538684 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-envoy\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.538886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.538706 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-data\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.538886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.538845 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-credential-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.538886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.538880 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-certs\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.539015 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.538961 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-token\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.539064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.539017 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gfgj\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-kube-api-access-9gfgj\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.539155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.539131 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istiod-ca-cert\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.539246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.539228 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-podinfo\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641061 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641025 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gfgj\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-kube-api-access-9gfgj\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641243 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641100 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istiod-ca-cert\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641243 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641150 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-podinfo\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641243 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641205 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641243 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641225 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-envoy\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641253 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-data\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641341 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-credential-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641389 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-certs\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641536 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641496 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-token\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641712 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641876 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641848 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-certs\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641946 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641885 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-credential-socket\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.641994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641971 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-data\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.642041 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.641994 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istiod-ca-cert\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.643856 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.643829 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-envoy\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.644075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.644059 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-podinfo\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.660778 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.660755 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-token\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.660859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.660814 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gfgj\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-kube-api-access-9gfgj\") pod \"e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:44.770744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:44.770677 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:45.064826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:45.064783 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c85ed67-3c59-4379-aba4-188f55d9bc92" path="/var/lib/kubelet/pods/3c85ed67-3c59-4379-aba4-188f55d9bc92/volumes" Oct 30 18:52:45.091066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:45.091037 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg"] Oct 30 18:52:45.094768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:45.094736 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:52:45.094864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:45.094801 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:52:45.094864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:45.094832 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:52:45.108467 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:45.108440 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" event={"ID":"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a","Type":"ContainerStarted","Data":"475afdf52929900af3919e48db3c6543e23c15a0b1fba8d762ecf103710e91eb"} Oct 30 18:52:46.172145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:46.171694 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" event={"ID":"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a","Type":"ContainerStarted","Data":"f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1"} Oct 30 18:52:46.245718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:46.245644 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" podStartSLOduration=2.245619583 podStartE2EDuration="2.245619583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:46.226715679 +0000 UTC m=+3265.275760593" watchObservedRunningTime="2026-10-30 18:52:46.245619583 +0000 UTC m=+3265.294664498" Oct 30 18:52:46.771243 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:46.771203 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:46.772654 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:46.772625 4862 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:52:46.772796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:46.772673 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 18:52:47.771101 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:47.771054 4862 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:52:47.771568 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:47.771118 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 18:52:48.771548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:48.771510 4862 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:52:48.771942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:48.771569 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 18:52:49.771930 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:49.771896 4862 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:52:49.772423 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:49.771951 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 18:52:49.880203 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:52:49.880163 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:52:50.771614 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:50.771579 4862 patch_prober.go:29] interesting pod/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body= Oct 30 18:52:50.771802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:50.771630 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" Oct 30 18:52:51.776434 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:51.776402 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:51.776865 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:51.776845 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:51.777620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:51.777597 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:52:52.323484 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.323445 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj"] Oct 30 18:52:52.328217 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.328193 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.343628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.343599 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c52ec7\"/\"openshift-service-ca.crt\"" Oct 30 18:52:52.345699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.345678 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj"] Oct 30 18:52:52.353059 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.353035 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c52ec7\"/\"kube-root-ca.crt\"" Oct 30 18:52:52.353162 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.353066 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c52ec7\"/\"default-dockercfg-b2htd\"" Oct 30 18:52:52.353162 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.353147 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c52ec7\"/\"test-model-c52ec7-kserve-self-signed-certs\"" Oct 30 18:52:52.490578 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.490537 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-tmp-dir\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.490578 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.490573 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-dshm\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.490844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.490617 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jh7\" (UniqueName: \"kubernetes.io/projected/b5fafa9f-cc81-422f-a01f-b381778d26a3-kube-api-access-76jh7\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.490844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.490794 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5fafa9f-cc81-422f-a01f-b381778d26a3-tls-certs\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.490977 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.490929 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-home\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.491054 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.491037 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-model-cache\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.592560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.592463 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-tmp-dir\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.593294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.593247 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-dshm\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.594005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.593982 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jh7\" (UniqueName: \"kubernetes.io/projected/b5fafa9f-cc81-422f-a01f-b381778d26a3-kube-api-access-76jh7\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.594234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.594213 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5fafa9f-cc81-422f-a01f-b381778d26a3-tls-certs\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.594834 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.593181 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-tmp-dir\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.595145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.595124 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-home\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.606061 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.606036 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-model-cache\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.606744 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.605169 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-home\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.606885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.606642 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-model-cache\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.634030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.633995 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jh7\" (UniqueName: \"kubernetes.io/projected/b5fafa9f-cc81-422f-a01f-b381778d26a3-kube-api-access-76jh7\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.635183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.635152 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5fafa9f-cc81-422f-a01f-b381778d26a3-tls-certs\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.647392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.646623 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-dshm\") pod \"test-model-c52ec7-kserve-766594dcfc-rm5vj\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:52.647392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:52.647093 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:52:53.036218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:53.036195 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj"] Oct 30 18:52:53.036588 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:52:53.036476 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5fafa9f_cc81_422f_a01f_b381778d26a3.slice/crio-bf8b72561dbe5b51cff557702b1afdb3dddcdde93a2cef29d562f20ea164aa0a WatchSource:0}: Error finding container bf8b72561dbe5b51cff557702b1afdb3dddcdde93a2cef29d562f20ea164aa0a: Status 404 returned error can't find the container with id bf8b72561dbe5b51cff557702b1afdb3dddcdde93a2cef29d562f20ea164aa0a Oct 30 18:52:53.260294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:53.259505 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" event={"ID":"b5fafa9f-cc81-422f-a01f-b381778d26a3","Type":"ContainerStarted","Data":"f7459ebbb195d2f7f713bd31a61c5038c27c377eb04db88a203f9f5424faa902"} Oct 30 18:52:53.260294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:53.259679 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" event={"ID":"b5fafa9f-cc81-422f-a01f-b381778d26a3","Type":"ContainerStarted","Data":"bf8b72561dbe5b51cff557702b1afdb3dddcdde93a2cef29d562f20ea164aa0a"} Oct 30 18:52:53.301819 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:53.301770 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" podStartSLOduration=1.301756498 podStartE2EDuration="1.301756498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:53.299553583 +0000 UTC m=+3272.348598496" watchObservedRunningTime="2026-10-30 18:52:53.301756498 +0000 UTC m=+3272.350801410" Oct 30 18:52:53.648489 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:52:53.648419 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:53:02.647819 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:02.647737 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:53:02.665113 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:02.665082 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:53:03.424382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:03.423486 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:53:06.483073 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.483038 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb"] Oct 30 18:53:06.489001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.488978 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.492012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.491988 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgw69\" (UniqueName: \"kubernetes.io/projected/9d34f511-440b-4a33-8b00-e22bf0e745db-kube-api-access-dgw69\") pod \"maas-api-revoke-sub-47b0b388-qkvvb\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.492347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.492320 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b0b388-qkvvb\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.492415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.492367 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b0b388-qkvvb\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.507305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.507278 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb"] Oct 30 18:53:06.513891 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.513867 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:53:06.514102 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.513963 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:53:06.594134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.594103 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b0b388-qkvvb\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.594134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.594139 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b0b388-qkvvb\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.594376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.594222 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgw69\" (UniqueName: \"kubernetes.io/projected/9d34f511-440b-4a33-8b00-e22bf0e745db-kube-api-access-dgw69\") pod \"maas-api-revoke-sub-47b0b388-qkvvb\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.595060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.595037 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b0b388-qkvvb\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.596881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.596850 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b0b388-qkvvb\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.613976 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.613952 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgw69\" (UniqueName: \"kubernetes.io/projected/9d34f511-440b-4a33-8b00-e22bf0e745db-kube-api-access-dgw69\") pod \"maas-api-revoke-sub-47b0b388-qkvvb\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:06.810341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:06.810315 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:07.158038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:07.158009 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb"] Oct 30 18:53:07.159696 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:53:07.159646 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d34f511_440b_4a33_8b00_e22bf0e745db.slice/crio-a9b690e73b38d5dfe8e9874b398159f22ae24beedf8585f08a5e7ab6c9e4e063 WatchSource:0}: Error finding container a9b690e73b38d5dfe8e9874b398159f22ae24beedf8585f08a5e7ab6c9e4e063: Status 404 returned error can't find the container with id a9b690e73b38d5dfe8e9874b398159f22ae24beedf8585f08a5e7ab6c9e4e063 Oct 30 18:53:07.460031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:07.459815 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" event={"ID":"9d34f511-440b-4a33-8b00-e22bf0e745db","Type":"ContainerStarted","Data":"a9b690e73b38d5dfe8e9874b398159f22ae24beedf8585f08a5e7ab6c9e4e063"} Oct 30 18:53:08.468909 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:08.468875 4862 generic.go:356] "Generic (PLEG): container finished" podID="9d34f511-440b-4a33-8b00-e22bf0e745db" containerID="a72aa390b48074d4f5b4de6bceb1d77c64260ae117f504d3641dcd67ecedab7a" exitCode=0 Oct 30 18:53:08.469416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:08.468968 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" event={"ID":"9d34f511-440b-4a33-8b00-e22bf0e745db","Type":"ContainerDied","Data":"a72aa390b48074d4f5b4de6bceb1d77c64260ae117f504d3641dcd67ecedab7a"} Oct 30 18:53:09.459528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.459494 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml"] Oct 30 18:53:09.464095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.464074 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:09.474880 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.474857 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml"] Oct 30 18:53:09.481724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.481701 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f37ca7\"" Oct 30 18:53:09.546295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.529387 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f37ca7-dockercfg-mtbrb\"" Oct 30 18:53:09.546295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.529443 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv"] Oct 30 18:53:09.595820 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.556030 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:09.704288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.622122 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpt5\" (UniqueName: \"kubernetes.io/projected/a3487105-c424-45bd-9913-3f90091b0e70-kube-api-access-7vpt5\") pod \"payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml\" (UID: \"a3487105-c424-45bd-9913-3f90091b0e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:09.704288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.622357 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3487105-c424-45bd-9913-3f90091b0e70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml\" (UID: \"a3487105-c424-45bd-9913-3f90091b0e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:09.764392 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.740637 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3487105-c424-45bd-9913-3f90091b0e70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml\" (UID: \"a3487105-c424-45bd-9913-3f90091b0e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:09.765993 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.764731 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv\" (UID: \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:09.766757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.766712 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpt5\" (UniqueName: \"kubernetes.io/projected/a3487105-c424-45bd-9913-3f90091b0e70-kube-api-access-7vpt5\") pod \"payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml\" (UID: \"a3487105-c424-45bd-9913-3f90091b0e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:09.767521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.767469 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5bt2\" (UniqueName: \"kubernetes.io/projected/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-kube-api-access-q5bt2\") pod \"payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv\" (UID: \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:09.776013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.775991 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv"] Oct 30 18:53:09.780302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.780231 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3487105-c424-45bd-9913-3f90091b0e70-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml\" (UID: \"a3487105-c424-45bd-9913-3f90091b0e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:09.803502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.803380 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpt5\" (UniqueName: \"kubernetes.io/projected/a3487105-c424-45bd-9913-3f90091b0e70-kube-api-access-7vpt5\") pod \"payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml\" (UID: \"a3487105-c424-45bd-9913-3f90091b0e70\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:09.869239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.869201 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5bt2\" (UniqueName: \"kubernetes.io/projected/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-kube-api-access-q5bt2\") pod \"payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv\" (UID: \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:09.869471 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.869431 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv\" (UID: \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:09.870516 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.870482 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv\" (UID: \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:09.903997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:09.902070 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5bt2\" (UniqueName: \"kubernetes.io/projected/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-kube-api-access-q5bt2\") pod \"payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv\" (UID: \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:10.001737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.001711 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:10.073848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.072449 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d34f511-440b-4a33-8b00-e22bf0e745db-kube-api-access-dgw69\" (UniqueName: \"kubernetes.io/projected/9d34f511-440b-4a33-8b00-e22bf0e745db-kube-api-access-dgw69\") pod \"9d34f511-440b-4a33-8b00-e22bf0e745db\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " Oct 30 18:53:10.073848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.072497 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-openshift-ca\") pod \"9d34f511-440b-4a33-8b00-e22bf0e745db\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " Oct 30 18:53:10.073848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.072632 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-xks-ca\") pod \"9d34f511-440b-4a33-8b00-e22bf0e745db\" (UID: \"9d34f511-440b-4a33-8b00-e22bf0e745db\") " Oct 30 18:53:10.073848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.073140 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-openshift-ca" pod "9d34f511-440b-4a33-8b00-e22bf0e745db" (UID: "9d34f511-440b-4a33-8b00-e22bf0e745db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:10.073848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.073363 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:10.078994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.078419 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d34f511-440b-4a33-8b00-e22bf0e745db-kube-api-access-dgw69" pod "9d34f511-440b-4a33-8b00-e22bf0e745db" (UID: "9d34f511-440b-4a33-8b00-e22bf0e745db"). InnerVolumeSpecName "kube-api-access-dgw69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:10.079738 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.079695 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-xks-ca" pod "9d34f511-440b-4a33-8b00-e22bf0e745db" (UID: "9d34f511-440b-4a33-8b00-e22bf0e745db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:10.088076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.088042 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:10.138658 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.138630 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:10.175664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.175633 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgw69\" (UniqueName: \"kubernetes.io/projected/9d34f511-440b-4a33-8b00-e22bf0e745db-kube-api-access-dgw69\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:10.175851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.175670 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d34f511-440b-4a33-8b00-e22bf0e745db-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:10.437166 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.437082 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml"] Oct 30 18:53:10.445554 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:53:10.444539 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3487105_c424_45bd_9913_3f90091b0e70.slice/crio-0aeca0c59cc1baa2a43c0f702a35a1ea160a5a41bbbfefcaad5df0d97d425c2b WatchSource:0}: Error finding container 0aeca0c59cc1baa2a43c0f702a35a1ea160a5a41bbbfefcaad5df0d97d425c2b: Status 404 returned error can't find the container with id 0aeca0c59cc1baa2a43c0f702a35a1ea160a5a41bbbfefcaad5df0d97d425c2b Oct 30 18:53:10.482273 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.482236 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv"] Oct 30 18:53:10.484839 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:53:10.484806 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb6ab44_54dc_4284_8e09_5d4a5d756d23.slice/crio-29a70e8f552c3c001f91f9014cef7bbe35021bea6f13a3bcf5152da0fe0b2e11 WatchSource:0}: Error finding container 29a70e8f552c3c001f91f9014cef7bbe35021bea6f13a3bcf5152da0fe0b2e11: Status 404 returned error can't find the container with id 29a70e8f552c3c001f91f9014cef7bbe35021bea6f13a3bcf5152da0fe0b2e11 Oct 30 18:53:10.493179 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.491850 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" event={"ID":"a3487105-c424-45bd-9913-3f90091b0e70","Type":"ContainerStarted","Data":"0aeca0c59cc1baa2a43c0f702a35a1ea160a5a41bbbfefcaad5df0d97d425c2b"} Oct 30 18:53:10.498312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.496484 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" Oct 30 18:53:10.498312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.496493 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb" event={"ID":"9d34f511-440b-4a33-8b00-e22bf0e745db","Type":"ContainerDied","Data":"a9b690e73b38d5dfe8e9874b398159f22ae24beedf8585f08a5e7ab6c9e4e063"} Oct 30 18:53:10.498312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:10.496544 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9b690e73b38d5dfe8e9874b398159f22ae24beedf8585f08a5e7ab6c9e4e063" Oct 30 18:53:11.503740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:11.503400 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:53:11.503740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:11.503457 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:11.511784 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:11.511749 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" event={"ID":"a3487105-c424-45bd-9913-3f90091b0e70","Type":"ContainerStarted","Data":"b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e"} Oct 30 18:53:11.511965 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:11.511859 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:11.513993 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:11.513972 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" event={"ID":"0cb6ab44-54dc-4284-8e09-5d4a5d756d23","Type":"ContainerStarted","Data":"74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8"} Oct 30 18:53:11.514097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:11.514017 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" event={"ID":"0cb6ab44-54dc-4284-8e09-5d4a5d756d23","Type":"ContainerStarted","Data":"29a70e8f552c3c001f91f9014cef7bbe35021bea6f13a3bcf5152da0fe0b2e11"} Oct 30 18:53:11.514097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:11.514051 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:11.551013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:11.550965 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" podStartSLOduration=2.5509510730000002 podStartE2EDuration="2.550951073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:11.549350664 +0000 UTC m=+3290.598395577" watchObservedRunningTime="2026-10-30 18:53:11.550951073 +0000 UTC m=+3290.599995991" Oct 30 18:53:11.590367 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:11.590316 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" podStartSLOduration=2.590301497 podStartE2EDuration="2.590301497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:11.587694914 +0000 UTC m=+3290.636739839" watchObservedRunningTime="2026-10-30 18:53:11.590301497 +0000 UTC m=+3290.639346409" Oct 30 18:53:14.539602 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:14.539571 4862 generic.go:356] "Generic (PLEG): container finished" podID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerID="3e56ebb0bd4644d0096aefd6d2a89031015022e90b0bd86b97f2a17deceb0510" exitCode=0 Oct 30 18:53:14.540079 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:14.539647 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" event={"ID":"6f8a371c-43aa-4fca-bf65-c22c553cbb8c","Type":"ContainerDied","Data":"3e56ebb0bd4644d0096aefd6d2a89031015022e90b0bd86b97f2a17deceb0510"} Oct 30 18:53:14.540129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:14.540078 4862 scope.go:122] "RemoveContainer" containerID="3e56ebb0bd4644d0096aefd6d2a89031015022e90b0bd86b97f2a17deceb0510" Oct 30 18:53:14.542236 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:14.542205 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:53:14.542375 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:14.542320 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:53:14.542451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:14.542381 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:53:15.576787 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:15.576733 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" event={"ID":"6f8a371c-43aa-4fca-bf65-c22c553cbb8c","Type":"ContainerStarted","Data":"84aef472547aa68285d3b6fece3a41aa2634e3fb5ebfd8eeea7c5f90f357d963"} Oct 30 18:53:15.902551 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:15.902479 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:53:15.902705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:15.902642 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:53:15.902705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:15.902688 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 18:53:16.902399 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:16.902365 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:53:16.902814 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:16.902419 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 18:53:17.157069 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.156970 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq"] Oct 30 18:53:17.157736 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.157719 4862 cpu_manager.go:414] "containerMap: removing container" podUID="9d34f511-440b-4a33-8b00-e22bf0e745db" containerName="revoke-keys" Oct 30 18:53:17.157819 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.157738 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d34f511-440b-4a33-8b00-e22bf0e745db" containerName="revoke-keys" Oct 30 18:53:17.157898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.157887 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d34f511-440b-4a33-8b00-e22bf0e745db" containerName="revoke-keys" Oct 30 18:53:17.162973 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.162945 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.169009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.168977 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq"] Oct 30 18:53:17.179151 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.179128 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:53:17.191133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.191105 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:53:17.305122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.305081 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9sm9\" (UniqueName: \"kubernetes.io/projected/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-kube-api-access-g9sm9\") pod \"maas-api-revoke-sub-489d6b99-849gq\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.305324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.305253 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489d6b99-849gq\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.305324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.305313 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489d6b99-849gq\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.406756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.406720 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489d6b99-849gq\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.406983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.406964 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9sm9\" (UniqueName: \"kubernetes.io/projected/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-kube-api-access-g9sm9\") pod \"maas-api-revoke-sub-489d6b99-849gq\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.407151 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.407104 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489d6b99-849gq\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.407485 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.407463 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-489d6b99-849gq\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.409695 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.409670 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-489d6b99-849gq\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.426764 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.426741 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9sm9\" (UniqueName: \"kubernetes.io/projected/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-kube-api-access-g9sm9\") pod \"maas-api-revoke-sub-489d6b99-849gq\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.480648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.480607 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:17.823089 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:53:17.823046 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd048b581_ed21_4e8d_bff2_d45ff2a72e1f.slice/crio-e8e5005797838dd8453eadc91689e28dec0331c72fb0cfba7cf39d5d7072849b WatchSource:0}: Error finding container e8e5005797838dd8453eadc91689e28dec0331c72fb0cfba7cf39d5d7072849b: Status 404 returned error can't find the container with id e8e5005797838dd8453eadc91689e28dec0331c72fb0cfba7cf39d5d7072849b Oct 30 18:53:17.824545 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.824520 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq"] Oct 30 18:53:17.902844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.902818 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:53:17.903149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:17.902872 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 18:53:18.608494 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:18.608454 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" event={"ID":"d048b581-ed21-4e8d-bff2-d45ff2a72e1f","Type":"ContainerStarted","Data":"f0d9bb79e5a5ba602404c68ffedcc8ecc17f44509633228c6ff228f6f72cc4d2"} Oct 30 18:53:18.608676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:18.608520 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" event={"ID":"d048b581-ed21-4e8d-bff2-d45ff2a72e1f","Type":"ContainerStarted","Data":"e8e5005797838dd8453eadc91689e28dec0331c72fb0cfba7cf39d5d7072849b"} Oct 30 18:53:18.657867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:18.657791 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" podStartSLOduration=1.6577722019999999 podStartE2EDuration="1.657772202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:18.65271323 +0000 UTC m=+3297.701758144" watchObservedRunningTime="2026-10-30 18:53:18.657772202 +0000 UTC m=+3297.706817116" Oct 30 18:53:18.902527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:18.902443 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:53:18.902527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:18.902499 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 18:53:19.658463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:19.658424 4862 generic.go:356] "Generic (PLEG): container finished" podID="d048b581-ed21-4e8d-bff2-d45ff2a72e1f" containerID="f0d9bb79e5a5ba602404c68ffedcc8ecc17f44509633228c6ff228f6f72cc4d2" exitCode=0 Oct 30 18:53:19.659520 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:19.659048 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" event={"ID":"d048b581-ed21-4e8d-bff2-d45ff2a72e1f","Type":"ContainerDied","Data":"f0d9bb79e5a5ba602404c68ffedcc8ecc17f44509633228c6ff228f6f72cc4d2"} Oct 30 18:53:19.880764 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:53:19.880730 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:53:19.902211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:19.902182 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:53:19.902361 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:19.902232 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.202:15021/healthz/ready\": dial tcp 10.133.0.202:15021: connect: connection refused" Oct 30 18:53:20.845648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.845615 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:20.899305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.899002 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-kube-api-access-g9sm9\" (UniqueName: \"kubernetes.io/projected/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-kube-api-access-g9sm9\") pod \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " Oct 30 18:53:20.899305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.899051 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-openshift-ca\") pod \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " Oct 30 18:53:20.899305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.899160 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-xks-ca\") pod \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\" (UID: \"d048b581-ed21-4e8d-bff2-d45ff2a72e1f\") " Oct 30 18:53:20.901430 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.900940 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-openshift-ca" pod "d048b581-ed21-4e8d-bff2-d45ff2a72e1f" (UID: "d048b581-ed21-4e8d-bff2-d45ff2a72e1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:20.903454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.903424 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-kube-api-access-g9sm9" pod "d048b581-ed21-4e8d-bff2-d45ff2a72e1f" (UID: "d048b581-ed21-4e8d-bff2-d45ff2a72e1f"). InnerVolumeSpecName "kube-api-access-g9sm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:20.904218 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.904196 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-xks-ca" pod "d048b581-ed21-4e8d-bff2-d45ff2a72e1f" (UID: "d048b581-ed21-4e8d-bff2-d45ff2a72e1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:20.971097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.971022 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:53:20.971678 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.971636 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:53:20.972656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:20.972631 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 18:53:21.001663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:21.001639 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9sm9\" (UniqueName: \"kubernetes.io/projected/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-kube-api-access-g9sm9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:21.001663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:21.001663 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:21.001807 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:21.001673 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d048b581-ed21-4e8d-bff2-d45ff2a72e1f-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:21.683180 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:21.683150 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" Oct 30 18:53:21.683532 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:21.683501 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq" event={"ID":"d048b581-ed21-4e8d-bff2-d45ff2a72e1f","Type":"ContainerDied","Data":"e8e5005797838dd8453eadc91689e28dec0331c72fb0cfba7cf39d5d7072849b"} Oct 30 18:53:21.683680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:21.683543 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8e5005797838dd8453eadc91689e28dec0331c72fb0cfba7cf39d5d7072849b" Oct 30 18:53:21.694548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:21.694518 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:53:21.697453 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:21.697428 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:53:22.435081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:22.435047 4862 scope.go:122] "RemoveContainer" containerID="4dbbf2d8c58b350a1810ea01f8401eeb885b957ca65dc2e813bbab31628df4e1" Oct 30 18:53:22.448118 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:22.448094 4862 scope.go:122] "RemoveContainer" containerID="25503e31ecd3bb508e57ba3de821e95cb12941e8063a0ffcb6aaf22433477b05" Oct 30 18:53:22.458257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:22.458238 4862 scope.go:122] "RemoveContainer" containerID="85e347463401de5fb543630e356ed745c8e4440a843f4a025d973d396cd50e45" Oct 30 18:53:22.526337 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:22.526313 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 18:53:22.526934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:22.526914 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 18:53:29.053548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.053515 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn"] Oct 30 18:53:29.054225 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.054210 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d048b581-ed21-4e8d-bff2-d45ff2a72e1f" containerName="revoke-keys" Oct 30 18:53:29.054310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.054226 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d048b581-ed21-4e8d-bff2-d45ff2a72e1f" containerName="revoke-keys" Oct 30 18:53:29.054421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.054409 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d048b581-ed21-4e8d-bff2-d45ff2a72e1f" containerName="revoke-keys" Oct 30 18:53:29.059833 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.059807 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.077858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.077830 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn"] Oct 30 18:53:29.141252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.141216 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.141508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.141469 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.141662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.141592 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.141797 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.141771 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq2qq\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-kube-api-access-xq2qq\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.242929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.242900 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.243078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.243013 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.243078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.243036 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.243165 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.243135 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq2qq\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-kube-api-access-xq2qq\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.243698 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.243672 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.243854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.243835 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.245714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.245696 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.262490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.262467 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq2qq\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-kube-api-access-xq2qq\") pod \"payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.381334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.378537 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:29.424418 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.424385 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb"] Oct 30 18:53:29.435425 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.435399 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.471308 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.463173 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:53:29.472364 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.472336 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:53:29.474172 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.474144 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7786d5e7-wszfb\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.474617 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.474593 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwc9w\" (UniqueName: \"kubernetes.io/projected/c0c9e4a0-c531-4985-ab09-54895977c7dc-kube-api-access-wwc9w\") pod \"maas-api-revoke-sub-7786d5e7-wszfb\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.478982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.478960 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7786d5e7-wszfb\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.494464 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.492689 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb"] Oct 30 18:53:29.603102 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.603070 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7786d5e7-wszfb\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.603401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.603381 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwc9w\" (UniqueName: \"kubernetes.io/projected/c0c9e4a0-c531-4985-ab09-54895977c7dc-kube-api-access-wwc9w\") pod \"maas-api-revoke-sub-7786d5e7-wszfb\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.604690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.604526 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7786d5e7-wszfb\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.605857 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.605810 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7786d5e7-wszfb\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.606296 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.606252 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7786d5e7-wszfb\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.629028 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.628365 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwc9w\" (UniqueName: \"kubernetes.io/projected/c0c9e4a0-c531-4985-ab09-54895977c7dc-kube-api-access-wwc9w\") pod \"maas-api-revoke-sub-7786d5e7-wszfb\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.841366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.841337 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:29.878324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.877885 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn"] Oct 30 18:53:29.881169 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:53:29.881143 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a62bd2_9993_4188_9d00_54cfff26d2f0.slice/crio-fd1170c397129c4641cd718c562986fc239a23e8c76ffa028892f3e5ba3f377b WatchSource:0}: Error finding container fd1170c397129c4641cd718c562986fc239a23e8c76ffa028892f3e5ba3f377b: Status 404 returned error can't find the container with id fd1170c397129c4641cd718c562986fc239a23e8c76ffa028892f3e5ba3f377b Oct 30 18:53:29.899457 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.899422 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v"] Oct 30 18:53:29.906691 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.906667 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:29.928254 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:29.928228 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v"] Oct 30 18:53:30.069490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.069456 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.069938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.069568 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.069938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.069766 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.069938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.069832 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2gcc\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-kube-api-access-b2gcc\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.171668 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.171640 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.171806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.171713 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.171891 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.171873 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.171958 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.171940 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2gcc\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-kube-api-access-b2gcc\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.172556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.172533 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.172693 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.172605 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.173363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.173340 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb"] Oct 30 18:53:30.173644 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:53:30.173625 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c9e4a0_c531_4985_ab09_54895977c7dc.slice/crio-e41ec2d731a861e2b412fd1f51260a75d50228aa833275f7e668eb3ccff418c9 WatchSource:0}: Error finding container e41ec2d731a861e2b412fd1f51260a75d50228aa833275f7e668eb3ccff418c9: Status 404 returned error can't find the container with id e41ec2d731a861e2b412fd1f51260a75d50228aa833275f7e668eb3ccff418c9 Oct 30 18:53:30.174380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.174351 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.194740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.194718 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2gcc\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-kube-api-access-b2gcc\") pod \"payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.243715 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.243691 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.550669 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:53:30.550643 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31133e8f_0935_4d1c_9d0f_c253c3244094.slice/crio-8639b16f113f280cdc11010a1a774143765d1e9a8dd0a759c56da0389f46a4d2 WatchSource:0}: Error finding container 8639b16f113f280cdc11010a1a774143765d1e9a8dd0a759c56da0389f46a4d2: Status 404 returned error can't find the container with id 8639b16f113f280cdc11010a1a774143765d1e9a8dd0a759c56da0389f46a4d2 Oct 30 18:53:30.552200 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.552166 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v"] Oct 30 18:53:30.877565 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.877532 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" event={"ID":"11a62bd2-9993-4188-9d00-54cfff26d2f0","Type":"ContainerStarted","Data":"871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05"} Oct 30 18:53:30.877758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.877621 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" event={"ID":"11a62bd2-9993-4188-9d00-54cfff26d2f0","Type":"ContainerStarted","Data":"fd1170c397129c4641cd718c562986fc239a23e8c76ffa028892f3e5ba3f377b"} Oct 30 18:53:30.877947 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.877928 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:30.879877 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.879841 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" event={"ID":"31133e8f-0935-4d1c-9d0f-c253c3244094","Type":"ContainerStarted","Data":"114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d"} Oct 30 18:53:30.880014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.879934 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" event={"ID":"31133e8f-0935-4d1c-9d0f-c253c3244094","Type":"ContainerStarted","Data":"8639b16f113f280cdc11010a1a774143765d1e9a8dd0a759c56da0389f46a4d2"} Oct 30 18:53:30.880123 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.880105 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:30.881760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.881735 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" event={"ID":"c0c9e4a0-c531-4985-ab09-54895977c7dc","Type":"ContainerStarted","Data":"730b3d4699fe1e12e4030268d9dfc2087e2b417ee749ecfaf599446cf36ae01f"} Oct 30 18:53:30.881886 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.881809 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" event={"ID":"c0c9e4a0-c531-4985-ab09-54895977c7dc","Type":"ContainerStarted","Data":"e41ec2d731a861e2b412fd1f51260a75d50228aa833275f7e668eb3ccff418c9"} Oct 30 18:53:30.918283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.918226 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" podStartSLOduration=1.918213564 podStartE2EDuration="1.918213564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:30.91612158 +0000 UTC m=+3309.965166503" watchObservedRunningTime="2026-10-30 18:53:30.918213564 +0000 UTC m=+3309.967258476" Oct 30 18:53:30.963597 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.963556 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" podStartSLOduration=1.963544648 podStartE2EDuration="1.963544648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:30.960065739 +0000 UTC m=+3310.009110654" watchObservedRunningTime="2026-10-30 18:53:30.963544648 +0000 UTC m=+3310.012589560" Oct 30 18:53:30.999079 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:30.999035 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" podStartSLOduration=1.999025306 podStartE2EDuration="1.999025306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:30.997844993 +0000 UTC m=+3310.046889906" watchObservedRunningTime="2026-10-30 18:53:30.999025306 +0000 UTC m=+3310.048070241" Oct 30 18:53:31.892037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:31.891999 4862 generic.go:356] "Generic (PLEG): container finished" podID="c0c9e4a0-c531-4985-ab09-54895977c7dc" containerID="730b3d4699fe1e12e4030268d9dfc2087e2b417ee749ecfaf599446cf36ae01f" exitCode=0 Oct 30 18:53:31.892689 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:31.892050 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" event={"ID":"c0c9e4a0-c531-4985-ab09-54895977c7dc","Type":"ContainerDied","Data":"730b3d4699fe1e12e4030268d9dfc2087e2b417ee749ecfaf599446cf36ae01f"} Oct 30 18:53:33.070323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.070293 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:33.112313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.112285 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-openshift-ca\") pod \"c0c9e4a0-c531-4985-ab09-54895977c7dc\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " Oct 30 18:53:33.112468 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.112422 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c9e4a0-c531-4985-ab09-54895977c7dc-kube-api-access-wwc9w\" (UniqueName: \"kubernetes.io/projected/c0c9e4a0-c531-4985-ab09-54895977c7dc-kube-api-access-wwc9w\") pod \"c0c9e4a0-c531-4985-ab09-54895977c7dc\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " Oct 30 18:53:33.112534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.112514 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-openshift-ca" pod "c0c9e4a0-c531-4985-ab09-54895977c7dc" (UID: "c0c9e4a0-c531-4985-ab09-54895977c7dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:33.112587 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.112519 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-xks-ca\") pod \"c0c9e4a0-c531-4985-ab09-54895977c7dc\" (UID: \"c0c9e4a0-c531-4985-ab09-54895977c7dc\") " Oct 30 18:53:33.113463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.113440 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:33.114829 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.114805 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-xks-ca" pod "c0c9e4a0-c531-4985-ab09-54895977c7dc" (UID: "c0c9e4a0-c531-4985-ab09-54895977c7dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:33.115182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.115155 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c9e4a0-c531-4985-ab09-54895977c7dc-kube-api-access-wwc9w" pod "c0c9e4a0-c531-4985-ab09-54895977c7dc" (UID: "c0c9e4a0-c531-4985-ab09-54895977c7dc"). InnerVolumeSpecName "kube-api-access-wwc9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:33.215158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.215097 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwc9w\" (UniqueName: \"kubernetes.io/projected/c0c9e4a0-c531-4985-ab09-54895977c7dc-kube-api-access-wwc9w\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:33.215158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.215123 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9e4a0-c531-4985-ab09-54895977c7dc-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:33.907755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.907725 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" Oct 30 18:53:33.907755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.907737 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb" event={"ID":"c0c9e4a0-c531-4985-ab09-54895977c7dc","Type":"ContainerDied","Data":"e41ec2d731a861e2b412fd1f51260a75d50228aa833275f7e668eb3ccff418c9"} Oct 30 18:53:33.907962 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:33.907813 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e41ec2d731a861e2b412fd1f51260a75d50228aa833275f7e668eb3ccff418c9" Oct 30 18:53:41.896317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:41.896251 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:53:41.898189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:41.898163 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:53:41.974413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:41.974247 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr"] Oct 30 18:53:42.034169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:42.034132 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd"] Oct 30 18:53:46.975488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:46.975433 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" podUID="2274b699-f5b5-4ef4-9660-1690e8c68381" containerName="payload-processing" containerID="cri-o://1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332" gracePeriod=25 Oct 30 18:53:47.034516 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.034486 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" podUID="9161c1fc-5591-4b85-97c9-321063263979" containerName="payload-processing" containerID="cri-o://411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115" gracePeriod=25 Oct 30 18:53:47.376301 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.374780 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:53:47.401649 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.401621 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-kube-api-access-f7lg5\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-kube-api-access-f7lg5\") pod \"2274b699-f5b5-4ef4-9660-1690e8c68381\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " Oct 30 18:53:47.401774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.401700 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-provider-credentials\") pod \"2274b699-f5b5-4ef4-9660-1690e8c68381\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " Oct 30 18:53:47.401774 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.401764 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-routing-overlay-volume\") pod \"2274b699-f5b5-4ef4-9660-1690e8c68381\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " Oct 30 18:53:47.401894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.401884 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-plugins-config-volume\") pod \"2274b699-f5b5-4ef4-9660-1690e8c68381\" (UID: \"2274b699-f5b5-4ef4-9660-1690e8c68381\") " Oct 30 18:53:47.402910 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.402876 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-routing-overlay-volume" pod "2274b699-f5b5-4ef4-9660-1690e8c68381" (UID: "2274b699-f5b5-4ef4-9660-1690e8c68381"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.403380 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.403350 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-plugins-config-volume" pod "2274b699-f5b5-4ef4-9660-1690e8c68381" (UID: "2274b699-f5b5-4ef4-9660-1690e8c68381"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.404499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.404472 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-kube-api-access-f7lg5" pod "2274b699-f5b5-4ef4-9660-1690e8c68381" (UID: "2274b699-f5b5-4ef4-9660-1690e8c68381"). InnerVolumeSpecName "kube-api-access-f7lg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:47.406612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.406589 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-provider-credentials" pod "2274b699-f5b5-4ef4-9660-1690e8c68381" (UID: "2274b699-f5b5-4ef4-9660-1690e8c68381"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:47.431543 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.431517 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:53:47.503813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.503786 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-routing-overlay-volume\") pod \"9161c1fc-5591-4b85-97c9-321063263979\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " Oct 30 18:53:47.503984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.503863 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-provider-credentials\") pod \"9161c1fc-5591-4b85-97c9-321063263979\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " Oct 30 18:53:47.503984 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.503955 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-plugins-config-volume\") pod \"9161c1fc-5591-4b85-97c9-321063263979\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " Oct 30 18:53:47.504105 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.503992 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-kube-api-access-25dp8\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-kube-api-access-25dp8\") pod \"9161c1fc-5591-4b85-97c9-321063263979\" (UID: \"9161c1fc-5591-4b85-97c9-321063263979\") " Oct 30 18:53:47.504238 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.504210 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-routing-overlay-volume" pod "9161c1fc-5591-4b85-97c9-321063263979" (UID: "9161c1fc-5591-4b85-97c9-321063263979"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.504701 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.504674 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-plugins-config-volume" pod "9161c1fc-5591-4b85-97c9-321063263979" (UID: "9161c1fc-5591-4b85-97c9-321063263979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.505285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.505247 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.505487 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.505468 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7lg5\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-kube-api-access-f7lg5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.505599 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.505585 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2274b699-f5b5-4ef4-9660-1690e8c68381-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.505697 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.505684 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.505795 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.505783 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9161c1fc-5591-4b85-97c9-321063263979-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.505889 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.505878 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2274b699-f5b5-4ef4-9660-1690e8c68381-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.506795 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.506769 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-provider-credentials" pod "9161c1fc-5591-4b85-97c9-321063263979" (UID: "9161c1fc-5591-4b85-97c9-321063263979"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:47.506887 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.506867 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-kube-api-access-25dp8" pod "9161c1fc-5591-4b85-97c9-321063263979" (UID: "9161c1fc-5591-4b85-97c9-321063263979"). InnerVolumeSpecName "kube-api-access-25dp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:47.607400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.607320 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25dp8\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-kube-api-access-25dp8\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.607400 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:47.607355 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9161c1fc-5591-4b85-97c9-321063263979-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.036949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.036906 4862 generic.go:356] "Generic (PLEG): container finished" podID="2274b699-f5b5-4ef4-9660-1690e8c68381" containerID="1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332" exitCode=0 Oct 30 18:53:48.037480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.036966 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" Oct 30 18:53:48.037480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.036997 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" event={"ID":"2274b699-f5b5-4ef4-9660-1690e8c68381","Type":"ContainerDied","Data":"1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332"} Oct 30 18:53:48.037480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.037081 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr" event={"ID":"2274b699-f5b5-4ef4-9660-1690e8c68381","Type":"ContainerDied","Data":"6bf4589ff21ee41ef84a4ef38763b5b7b55ee6bcdefc67d0131db440e81941e7"} Oct 30 18:53:48.037480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.037123 4862 scope.go:122] "RemoveContainer" containerID="1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332" Oct 30 18:53:48.039421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.038833 4862 generic.go:356] "Generic (PLEG): container finished" podID="9161c1fc-5591-4b85-97c9-321063263979" containerID="411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115" exitCode=0 Oct 30 18:53:48.039421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.039094 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" event={"ID":"9161c1fc-5591-4b85-97c9-321063263979","Type":"ContainerDied","Data":"411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115"} Oct 30 18:53:48.039421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.039137 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" event={"ID":"9161c1fc-5591-4b85-97c9-321063263979","Type":"ContainerDied","Data":"37a1516ac078b2114343df4c1680d82ed01418c556f7ce940ae41d8769af42df"} Oct 30 18:53:48.039421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.039225 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd" Oct 30 18:53:48.052530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.051902 4862 scope.go:122] "RemoveContainer" containerID="1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332" Oct 30 18:53:48.052530 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:53:48.052226 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332\": container with ID starting with 1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332 not found: ID does not exist" containerID="1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332" Oct 30 18:53:48.052530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.052254 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332"} err="failed to get container status \"1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332\": rpc error: code = NotFound desc = could not find container \"1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332\": container with ID starting with 1b5599ad69699987b383426ab02d2afc436c99dcbacba57f4f0f54d8f19c5332 not found: ID does not exist" Oct 30 18:53:48.052530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.052287 4862 scope.go:122] "RemoveContainer" containerID="411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115" Oct 30 18:53:48.059169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.059144 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:53:48.063558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.063542 4862 scope.go:122] "RemoveContainer" containerID="411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115" Oct 30 18:53:48.063884 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:53:48.063865 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115\": container with ID starting with 411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115 not found: ID does not exist" containerID="411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115" Oct 30 18:53:48.063943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.063891 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115"} err="failed to get container status \"411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115\": rpc error: code = NotFound desc = could not find container \"411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115\": container with ID starting with 411558b4107d9be2e5d7382febfd7834c428a44ae7705902035e5c3acda56115 not found: ID does not exist" Oct 30 18:53:48.093861 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.093835 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr"] Oct 30 18:53:48.099555 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.099532 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-7f8f2xpvr"] Oct 30 18:53:48.136645 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.136612 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd"] Oct 30 18:53:48.140427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:48.140401 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-76df7rctd"] Oct 30 18:53:49.065006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:49.064970 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2274b699-f5b5-4ef4-9660-1690e8c68381" path="/var/lib/kubelet/pods/2274b699-f5b5-4ef4-9660-1690e8c68381/volumes" Oct 30 18:53:49.065639 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:49.065463 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9161c1fc-5591-4b85-97c9-321063263979" path="/var/lib/kubelet/pods/9161c1fc-5591-4b85-97c9-321063263979/volumes" Oct 30 18:53:49.875677 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:53:49.875636 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:53:50.955071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.955040 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k"] Oct 30 18:53:50.955941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.955917 4862 cpu_manager.go:414] "containerMap: removing container" podUID="9161c1fc-5591-4b85-97c9-321063263979" containerName="payload-processing" Oct 30 18:53:50.955941 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.955941 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9161c1fc-5591-4b85-97c9-321063263979" containerName="payload-processing" Oct 30 18:53:50.956150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.956002 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c0c9e4a0-c531-4985-ab09-54895977c7dc" containerName="revoke-keys" Oct 30 18:53:50.956150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.956013 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c9e4a0-c531-4985-ab09-54895977c7dc" containerName="revoke-keys" Oct 30 18:53:50.956150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.956051 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2274b699-f5b5-4ef4-9660-1690e8c68381" containerName="payload-processing" Oct 30 18:53:50.956150 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.956058 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2274b699-f5b5-4ef4-9660-1690e8c68381" containerName="payload-processing" Oct 30 18:53:50.956422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.956246 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2274b699-f5b5-4ef4-9660-1690e8c68381" containerName="payload-processing" Oct 30 18:53:50.956422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.956287 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="9161c1fc-5591-4b85-97c9-321063263979" containerName="payload-processing" Oct 30 18:53:50.956422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.956308 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c9e4a0-c531-4985-ab09-54895977c7dc" containerName="revoke-keys" Oct 30 18:53:50.961658 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.961635 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:50.976388 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.976364 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:53:50.979081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.979058 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k"] Oct 30 18:53:50.986323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:50.986302 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:53:51.077576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.077550 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382800df-4182-49aa-89e9-1ac37997127f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976b4c8c-6586k\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.077746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.077610 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nq8\" (UniqueName: \"kubernetes.io/projected/382800df-4182-49aa-89e9-1ac37997127f-kube-api-access-89nq8\") pod \"maas-api-revoke-sub-976b4c8c-6586k\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.077746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.077698 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382800df-4182-49aa-89e9-1ac37997127f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976b4c8c-6586k\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.178795 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.178766 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382800df-4182-49aa-89e9-1ac37997127f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976b4c8c-6586k\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.179232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.179200 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382800df-4182-49aa-89e9-1ac37997127f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976b4c8c-6586k\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.179457 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.179437 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nq8\" (UniqueName: \"kubernetes.io/projected/382800df-4182-49aa-89e9-1ac37997127f-kube-api-access-89nq8\") pod \"maas-api-revoke-sub-976b4c8c-6586k\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.179588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.179566 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382800df-4182-49aa-89e9-1ac37997127f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976b4c8c-6586k\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.181779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.181756 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382800df-4182-49aa-89e9-1ac37997127f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976b4c8c-6586k\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.198356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.198325 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nq8\" (UniqueName: \"kubernetes.io/projected/382800df-4182-49aa-89e9-1ac37997127f-kube-api-access-89nq8\") pod \"maas-api-revoke-sub-976b4c8c-6586k\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.275792 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.275728 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:51.614075 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:53:51.614041 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382800df_4182_49aa_89e9_1ac37997127f.slice/crio-3297d6875cd862251b6d7471fe2cc2c5a9d3546a15fdb5ae98c85017c763bc6e WatchSource:0}: Error finding container 3297d6875cd862251b6d7471fe2cc2c5a9d3546a15fdb5ae98c85017c763bc6e: Status 404 returned error can't find the container with id 3297d6875cd862251b6d7471fe2cc2c5a9d3546a15fdb5ae98c85017c763bc6e Oct 30 18:53:51.614981 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:51.614939 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k"] Oct 30 18:53:52.075346 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:52.075298 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" event={"ID":"382800df-4182-49aa-89e9-1ac37997127f","Type":"ContainerStarted","Data":"edcafaa6cdc3866f942af0ff8bf193e36993986aa523385694bd58906409f4fd"} Oct 30 18:53:52.075815 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:52.075375 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" event={"ID":"382800df-4182-49aa-89e9-1ac37997127f","Type":"ContainerStarted","Data":"3297d6875cd862251b6d7471fe2cc2c5a9d3546a15fdb5ae98c85017c763bc6e"} Oct 30 18:53:52.119216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:52.119168 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" podStartSLOduration=2.119156364 podStartE2EDuration="2.119156364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:52.111513312 +0000 UTC m=+3331.160558224" watchObservedRunningTime="2026-10-30 18:53:52.119156364 +0000 UTC m=+3331.168201277" Oct 30 18:53:52.693867 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:53:52.693841 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382800df_4182_49aa_89e9_1ac37997127f.slice/crio-edcafaa6cdc3866f942af0ff8bf193e36993986aa523385694bd58906409f4fd.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:53:52.828504 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:52.828462 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp"] Oct 30 18:53:52.833075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:52.833054 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:52.851415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:52.851389 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp"] Oct 30 18:53:52.909030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:52.908960 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abc57551-6ctmp\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:52.909174 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:52.909110 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abc57551-6ctmp\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:52.909306 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:52.909285 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvq4c\" (UniqueName: \"kubernetes.io/projected/01eb988c-0f90-4346-bca3-af9a971ecd56-kube-api-access-gvq4c\") pod \"maas-api-revoke-sub-abc57551-6ctmp\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:53.010712 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.010683 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abc57551-6ctmp\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:53.010847 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.010834 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvq4c\" (UniqueName: \"kubernetes.io/projected/01eb988c-0f90-4346-bca3-af9a971ecd56-kube-api-access-gvq4c\") pod \"maas-api-revoke-sub-abc57551-6ctmp\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:53.010938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.010918 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abc57551-6ctmp\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:53.011428 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.011407 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abc57551-6ctmp\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:53.013437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.013417 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abc57551-6ctmp\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:53.029953 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.029932 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvq4c\" (UniqueName: \"kubernetes.io/projected/01eb988c-0f90-4346-bca3-af9a971ecd56-kube-api-access-gvq4c\") pod \"maas-api-revoke-sub-abc57551-6ctmp\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:53.085043 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.085015 4862 generic.go:356] "Generic (PLEG): container finished" podID="382800df-4182-49aa-89e9-1ac37997127f" containerID="edcafaa6cdc3866f942af0ff8bf193e36993986aa523385694bd58906409f4fd" exitCode=0 Oct 30 18:53:53.085422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.085109 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" event={"ID":"382800df-4182-49aa-89e9-1ac37997127f","Type":"ContainerDied","Data":"edcafaa6cdc3866f942af0ff8bf193e36993986aa523385694bd58906409f4fd"} Oct 30 18:53:53.152312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.152286 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:53.469802 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:53:53.469766 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01eb988c_0f90_4346_bca3_af9a971ecd56.slice/crio-4f20b029e12932651bd2c728eb978e83722b59d64c2febcf49c9278b0bebc048 WatchSource:0}: Error finding container 4f20b029e12932651bd2c728eb978e83722b59d64c2febcf49c9278b0bebc048: Status 404 returned error can't find the container with id 4f20b029e12932651bd2c728eb978e83722b59d64c2febcf49c9278b0bebc048 Oct 30 18:53:53.470075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:53.470049 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp"] Oct 30 18:53:54.094596 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.094561 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" event={"ID":"01eb988c-0f90-4346-bca3-af9a971ecd56","Type":"ContainerStarted","Data":"bd8a3b36052d3b312dd6057f766142301df22f6f03947686ee7a4cda007ebc72"} Oct 30 18:53:54.094994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.094613 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" event={"ID":"01eb988c-0f90-4346-bca3-af9a971ecd56","Type":"ContainerStarted","Data":"4f20b029e12932651bd2c728eb978e83722b59d64c2febcf49c9278b0bebc048"} Oct 30 18:53:54.137921 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.137869 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" podStartSLOduration=2.137853634 podStartE2EDuration="2.137853634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:54.134078051 +0000 UTC m=+3333.183122965" watchObservedRunningTime="2026-10-30 18:53:54.137853634 +0000 UTC m=+3333.186898547" Oct 30 18:53:54.265055 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.265030 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:54.439410 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.439379 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382800df-4182-49aa-89e9-1ac37997127f-kube-api-access-89nq8\" (UniqueName: \"kubernetes.io/projected/382800df-4182-49aa-89e9-1ac37997127f-kube-api-access-89nq8\") pod \"382800df-4182-49aa-89e9-1ac37997127f\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " Oct 30 18:53:54.439554 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.439533 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382800df-4182-49aa-89e9-1ac37997127f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382800df-4182-49aa-89e9-1ac37997127f-maas-api-xks-ca\") pod \"382800df-4182-49aa-89e9-1ac37997127f\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " Oct 30 18:53:54.439650 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.439633 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382800df-4182-49aa-89e9-1ac37997127f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382800df-4182-49aa-89e9-1ac37997127f-maas-api-openshift-ca\") pod \"382800df-4182-49aa-89e9-1ac37997127f\" (UID: \"382800df-4182-49aa-89e9-1ac37997127f\") " Oct 30 18:53:54.439970 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.439948 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382800df-4182-49aa-89e9-1ac37997127f-maas-api-openshift-ca" pod "382800df-4182-49aa-89e9-1ac37997127f" (UID: "382800df-4182-49aa-89e9-1ac37997127f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:54.440294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.440275 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/382800df-4182-49aa-89e9-1ac37997127f-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:54.441595 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.441576 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382800df-4182-49aa-89e9-1ac37997127f-kube-api-access-89nq8" pod "382800df-4182-49aa-89e9-1ac37997127f" (UID: "382800df-4182-49aa-89e9-1ac37997127f"). InnerVolumeSpecName "kube-api-access-89nq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:54.441667 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.441646 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382800df-4182-49aa-89e9-1ac37997127f-maas-api-xks-ca" pod "382800df-4182-49aa-89e9-1ac37997127f" (UID: "382800df-4182-49aa-89e9-1ac37997127f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:54.541685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.541658 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89nq8\" (UniqueName: \"kubernetes.io/projected/382800df-4182-49aa-89e9-1ac37997127f-kube-api-access-89nq8\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:54.541685 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:54.541682 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/382800df-4182-49aa-89e9-1ac37997127f-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:55.102922 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:55.102887 4862 generic.go:356] "Generic (PLEG): container finished" podID="01eb988c-0f90-4346-bca3-af9a971ecd56" containerID="bd8a3b36052d3b312dd6057f766142301df22f6f03947686ee7a4cda007ebc72" exitCode=0 Oct 30 18:53:55.103328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:55.102969 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" event={"ID":"01eb988c-0f90-4346-bca3-af9a971ecd56","Type":"ContainerDied","Data":"bd8a3b36052d3b312dd6057f766142301df22f6f03947686ee7a4cda007ebc72"} Oct 30 18:53:55.104359 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:55.104337 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" event={"ID":"382800df-4182-49aa-89e9-1ac37997127f","Type":"ContainerDied","Data":"3297d6875cd862251b6d7471fe2cc2c5a9d3546a15fdb5ae98c85017c763bc6e"} Oct 30 18:53:55.104473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:55.104368 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k" Oct 30 18:53:55.104473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:55.104375 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3297d6875cd862251b6d7471fe2cc2c5a9d3546a15fdb5ae98c85017c763bc6e" Oct 30 18:53:56.270157 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.270129 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:56.367551 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.367477 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01eb988c-0f90-4346-bca3-af9a971ecd56-kube-api-access-gvq4c\" (UniqueName: \"kubernetes.io/projected/01eb988c-0f90-4346-bca3-af9a971ecd56-kube-api-access-gvq4c\") pod \"01eb988c-0f90-4346-bca3-af9a971ecd56\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " Oct 30 18:53:56.367722 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.367595 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-xks-ca\") pod \"01eb988c-0f90-4346-bca3-af9a971ecd56\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " Oct 30 18:53:56.367722 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.367637 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-openshift-ca\") pod \"01eb988c-0f90-4346-bca3-af9a971ecd56\" (UID: \"01eb988c-0f90-4346-bca3-af9a971ecd56\") " Oct 30 18:53:56.368050 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.368020 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-openshift-ca" pod "01eb988c-0f90-4346-bca3-af9a971ecd56" (UID: "01eb988c-0f90-4346-bca3-af9a971ecd56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:56.368249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.368226 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:56.369832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.369812 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-xks-ca" pod "01eb988c-0f90-4346-bca3-af9a971ecd56" (UID: "01eb988c-0f90-4346-bca3-af9a971ecd56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:56.369929 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.369869 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01eb988c-0f90-4346-bca3-af9a971ecd56-kube-api-access-gvq4c" pod "01eb988c-0f90-4346-bca3-af9a971ecd56" (UID: "01eb988c-0f90-4346-bca3-af9a971ecd56"). InnerVolumeSpecName "kube-api-access-gvq4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:56.469735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.469705 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvq4c\" (UniqueName: \"kubernetes.io/projected/01eb988c-0f90-4346-bca3-af9a971ecd56-kube-api-access-gvq4c\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:56.469735 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:56.469730 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01eb988c-0f90-4346-bca3-af9a971ecd56-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:53:57.121116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:57.121079 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" event={"ID":"01eb988c-0f90-4346-bca3-af9a971ecd56","Type":"ContainerDied","Data":"4f20b029e12932651bd2c728eb978e83722b59d64c2febcf49c9278b0bebc048"} Oct 30 18:53:57.121116 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:57.121110 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp" Oct 30 18:53:57.121312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:53:57.121140 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f20b029e12932651bd2c728eb978e83722b59d64c2febcf49c9278b0bebc048" Oct 30 18:54:02.574811 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:02.574773 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v"] Oct 30 18:54:03.495497 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:03.495460 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn"] Oct 30 18:54:04.262301 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.262246 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll"] Oct 30 18:54:04.263211 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.263189 4862 cpu_manager.go:414] "containerMap: removing container" podUID="01eb988c-0f90-4346-bca3-af9a971ecd56" containerName="revoke-keys" Oct 30 18:54:04.263382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.263212 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01eb988c-0f90-4346-bca3-af9a971ecd56" containerName="revoke-keys" Oct 30 18:54:04.263382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.263248 4862 cpu_manager.go:414] "containerMap: removing container" podUID="382800df-4182-49aa-89e9-1ac37997127f" containerName="revoke-keys" Oct 30 18:54:04.263382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.263256 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382800df-4182-49aa-89e9-1ac37997127f" containerName="revoke-keys" Oct 30 18:54:04.263561 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.263459 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="01eb988c-0f90-4346-bca3-af9a971ecd56" containerName="revoke-keys" Oct 30 18:54:04.263561 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.263484 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="382800df-4182-49aa-89e9-1ac37997127f" containerName="revoke-keys" Oct 30 18:54:04.268472 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.268451 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.285743 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.285714 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll"] Oct 30 18:54:04.293572 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.293536 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:54:04.293727 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.293638 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:54:04.327221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.327188 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f01aaf7-pw4ll\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.327466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.327445 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f01aaf7-pw4ll\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.327535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.327477 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v49q5\" (UniqueName: \"kubernetes.io/projected/a8f1b6ee-f340-428c-9287-f0f5630c3640-kube-api-access-v49q5\") pod \"maas-api-revoke-sub-7f01aaf7-pw4ll\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.429107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.429069 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f01aaf7-pw4ll\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.429305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.429122 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v49q5\" (UniqueName: \"kubernetes.io/projected/a8f1b6ee-f340-428c-9287-f0f5630c3640-kube-api-access-v49q5\") pod \"maas-api-revoke-sub-7f01aaf7-pw4ll\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.429305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.429288 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f01aaf7-pw4ll\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.429917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.429890 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f01aaf7-pw4ll\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.431862 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.431840 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f01aaf7-pw4ll\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.448468 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.448440 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v49q5\" (UniqueName: \"kubernetes.io/projected/a8f1b6ee-f340-428c-9287-f0f5630c3640-kube-api-access-v49q5\") pod \"maas-api-revoke-sub-7f01aaf7-pw4ll\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.590447 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.590411 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:04.904646 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:04.904615 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll"] Oct 30 18:54:04.905419 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:54:04.905391 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f1b6ee_f340_428c_9287_f0f5630c3640.slice/crio-4b5311ae999359b941b78d92b74062f717d3cda53c2e9f4c35d61fb3ab742f54 WatchSource:0}: Error finding container 4b5311ae999359b941b78d92b74062f717d3cda53c2e9f4c35d61fb3ab742f54: Status 404 returned error can't find the container with id 4b5311ae999359b941b78d92b74062f717d3cda53c2e9f4c35d61fb3ab742f54 Oct 30 18:54:05.196042 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:05.195955 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" event={"ID":"a8f1b6ee-f340-428c-9287-f0f5630c3640","Type":"ContainerStarted","Data":"4b5311ae999359b941b78d92b74062f717d3cda53c2e9f4c35d61fb3ab742f54"} Oct 30 18:54:06.206789 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.206703 4862 generic.go:356] "Generic (PLEG): container finished" podID="a8f1b6ee-f340-428c-9287-f0f5630c3640" containerID="bc83dec156b38eee07df6758b0a9615ba7719f94fb7abb5e744cbdbe8a5a0e51" exitCode=0 Oct 30 18:54:06.206789 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.206754 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" event={"ID":"a8f1b6ee-f340-428c-9287-f0f5630c3640","Type":"ContainerDied","Data":"bc83dec156b38eee07df6758b0a9615ba7719f94fb7abb5e744cbdbe8a5a0e51"} Oct 30 18:54:06.338896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.338864 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2"] Oct 30 18:54:06.343361 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.343344 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.361928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.361897 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f37ca7-serving-cert\"" Oct 30 18:54:06.365212 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.365182 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2"] Oct 30 18:54:06.466522 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.466432 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.466702 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.466675 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-metrics-certs\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.466781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.466766 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dmgh\" (UniqueName: \"kubernetes.io/projected/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-kube-api-access-7dmgh\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.466848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.466836 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-maas-api-tls\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.568943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.568583 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dmgh\" (UniqueName: \"kubernetes.io/projected/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-kube-api-access-7dmgh\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.569314 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.569288 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-maas-api-tls\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.569775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.569748 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.578734 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.570107 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-metrics-certs\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.579071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.573957 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.594421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.594390 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dmgh\" (UniqueName: \"kubernetes.io/projected/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-kube-api-access-7dmgh\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.598303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.595058 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-metrics-certs\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.598303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.596090 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-maas-api-tls\") pod \"maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.657832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.657800 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:06.971045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.971015 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2"] Oct 30 18:54:06.972071 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:54:06.972005 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37c63f2_e0ec_4f50_bc8e_98f9a0e19532.slice/crio-5d9ef88c9004617b30787e9c84d67ebf13d39a91d1b7ee12ead211d489772cf8 WatchSource:0}: Error finding container 5d9ef88c9004617b30787e9c84d67ebf13d39a91d1b7ee12ead211d489772cf8: Status 404 returned error can't find the container with id 5d9ef88c9004617b30787e9c84d67ebf13d39a91d1b7ee12ead211d489772cf8 Oct 30 18:54:06.974158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:06.974138 4862 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:54:07.033325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.033235 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn"] Oct 30 18:54:07.037455 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.037432 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056faee6-286mn"] Oct 30 18:54:07.063670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.063648 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82edc819-af71-490d-8eec-289d197b4469" path="/var/lib/kubelet/pods/82edc819-af71-490d-8eec-289d197b4469/volumes" Oct 30 18:54:07.222140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.222051 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" event={"ID":"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532","Type":"ContainerStarted","Data":"5d9ef88c9004617b30787e9c84d67ebf13d39a91d1b7ee12ead211d489772cf8"} Oct 30 18:54:07.487288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.483527 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:07.585056 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.575320 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" podUID="31133e8f-0935-4d1c-9d0f-c253c3244094" containerName="payload-processing" containerID="cri-o://114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d" gracePeriod=25 Oct 30 18:54:07.632768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.632734 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-openshift-ca\") pod \"a8f1b6ee-f340-428c-9287-f0f5630c3640\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " Oct 30 18:54:07.632935 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.632917 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-xks-ca\") pod \"a8f1b6ee-f340-428c-9287-f0f5630c3640\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " Oct 30 18:54:07.633118 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.633091 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f1b6ee-f340-428c-9287-f0f5630c3640-kube-api-access-v49q5\" (UniqueName: \"kubernetes.io/projected/a8f1b6ee-f340-428c-9287-f0f5630c3640-kube-api-access-v49q5\") pod \"a8f1b6ee-f340-428c-9287-f0f5630c3640\" (UID: \"a8f1b6ee-f340-428c-9287-f0f5630c3640\") " Oct 30 18:54:07.636938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.636909 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-openshift-ca" pod "a8f1b6ee-f340-428c-9287-f0f5630c3640" (UID: "a8f1b6ee-f340-428c-9287-f0f5630c3640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:07.649288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.647668 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-xks-ca" pod "a8f1b6ee-f340-428c-9287-f0f5630c3640" (UID: "a8f1b6ee-f340-428c-9287-f0f5630c3640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:07.650918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.650893 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f1b6ee-f340-428c-9287-f0f5630c3640-kube-api-access-v49q5" pod "a8f1b6ee-f340-428c-9287-f0f5630c3640" (UID: "a8f1b6ee-f340-428c-9287-f0f5630c3640"). InnerVolumeSpecName "kube-api-access-v49q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:07.736100 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.736068 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v49q5\" (UniqueName: \"kubernetes.io/projected/a8f1b6ee-f340-428c-9287-f0f5630c3640-kube-api-access-v49q5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:07.736238 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.736106 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:07.736238 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:07.736122 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f1b6ee-f340-428c-9287-f0f5630c3640-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:08.047005 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.046977 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:54:08.151241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.151214 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-kube-api-access-b2gcc\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-kube-api-access-b2gcc\") pod \"31133e8f-0935-4d1c-9d0f-c253c3244094\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " Oct 30 18:54:08.151414 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.151283 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-plugins-config-volume\") pod \"31133e8f-0935-4d1c-9d0f-c253c3244094\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " Oct 30 18:54:08.151414 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.151387 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-routing-overlay-volume\") pod \"31133e8f-0935-4d1c-9d0f-c253c3244094\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " Oct 30 18:54:08.151509 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.151493 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-provider-credentials\") pod \"31133e8f-0935-4d1c-9d0f-c253c3244094\" (UID: \"31133e8f-0935-4d1c-9d0f-c253c3244094\") " Oct 30 18:54:08.151768 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.151740 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-plugins-config-volume" pod "31133e8f-0935-4d1c-9d0f-c253c3244094" (UID: "31133e8f-0935-4d1c-9d0f-c253c3244094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:08.151900 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.151773 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-routing-overlay-volume" pod "31133e8f-0935-4d1c-9d0f-c253c3244094" (UID: "31133e8f-0935-4d1c-9d0f-c253c3244094"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:08.152636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.152604 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:08.152636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.152635 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31133e8f-0935-4d1c-9d0f-c253c3244094-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:08.153740 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.153719 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-kube-api-access-b2gcc" pod "31133e8f-0935-4d1c-9d0f-c253c3244094" (UID: "31133e8f-0935-4d1c-9d0f-c253c3244094"). InnerVolumeSpecName "kube-api-access-b2gcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:08.153867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.153844 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-provider-credentials" pod "31133e8f-0935-4d1c-9d0f-c253c3244094" (UID: "31133e8f-0935-4d1c-9d0f-c253c3244094"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:08.232425 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.232399 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" Oct 30 18:54:08.232915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.232432 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll" event={"ID":"a8f1b6ee-f340-428c-9287-f0f5630c3640","Type":"ContainerDied","Data":"4b5311ae999359b941b78d92b74062f717d3cda53c2e9f4c35d61fb3ab742f54"} Oct 30 18:54:08.232915 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.232508 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5311ae999359b941b78d92b74062f717d3cda53c2e9f4c35d61fb3ab742f54" Oct 30 18:54:08.254022 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.253982 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" event={"ID":"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532","Type":"ContainerStarted","Data":"fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b"} Oct 30 18:54:08.254223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.254077 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:08.261808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.261774 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:08.261952 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.261812 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2gcc\" (UniqueName: \"kubernetes.io/projected/31133e8f-0935-4d1c-9d0f-c253c3244094-kube-api-access-b2gcc\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:08.264748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.264697 4862 generic.go:356] "Generic (PLEG): container finished" podID="31133e8f-0935-4d1c-9d0f-c253c3244094" containerID="114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d" exitCode=0 Oct 30 18:54:08.264894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.264824 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" event={"ID":"31133e8f-0935-4d1c-9d0f-c253c3244094","Type":"ContainerDied","Data":"114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d"} Oct 30 18:54:08.264894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.264881 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" event={"ID":"31133e8f-0935-4d1c-9d0f-c253c3244094","Type":"ContainerDied","Data":"8639b16f113f280cdc11010a1a774143765d1e9a8dd0a759c56da0389f46a4d2"} Oct 30 18:54:08.265009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.264916 4862 scope.go:122] "RemoveContainer" containerID="114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d" Oct 30 18:54:08.265156 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.265080 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v" Oct 30 18:54:08.277994 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.277976 4862 scope.go:122] "RemoveContainer" containerID="114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d" Oct 30 18:54:08.278332 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:54:08.278308 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d\": container with ID starting with 114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d not found: ID does not exist" containerID="114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d" Oct 30 18:54:08.278406 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.278339 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d"} err="failed to get container status \"114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d\": rpc error: code = NotFound desc = could not find container \"114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d\": container with ID starting with 114700a7f252326e70504b88a0df9ca720d52276696c64907b5c69c3c1ddb95d not found: ID does not exist" Oct 30 18:54:08.298340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.298303 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" podStartSLOduration=1.7905011709999998 podStartE2EDuration="2.298288233s" totalImagesPullingTime="507.787062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:08.294370059 +0000 UTC m=+3347.343414975" watchObservedRunningTime="2026-10-30 18:54:08.298288233 +0000 UTC m=+3347.347333147" Oct 30 18:54:08.336017 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.335992 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v"] Oct 30 18:54:08.339857 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.339839 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-38e0db/payload-processing-external-model-e2e-extmod-b-38e0db-95d77656v"] Oct 30 18:54:08.496806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.496765 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" podUID="11a62bd2-9993-4188-9d00-54cfff26d2f0" containerName="payload-processing" containerID="cri-o://871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05" gracePeriod=25 Oct 30 18:54:08.849837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.849770 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:54:08.979008 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.978974 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-provider-credentials\") pod \"11a62bd2-9993-4188-9d00-54cfff26d2f0\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " Oct 30 18:54:08.979148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.979041 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-plugins-config-volume\") pod \"11a62bd2-9993-4188-9d00-54cfff26d2f0\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " Oct 30 18:54:08.979148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.979113 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-routing-overlay-volume\") pod \"11a62bd2-9993-4188-9d00-54cfff26d2f0\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " Oct 30 18:54:08.979488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.979466 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-plugins-config-volume" pod "11a62bd2-9993-4188-9d00-54cfff26d2f0" (UID: "11a62bd2-9993-4188-9d00-54cfff26d2f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:08.979573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.979526 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-routing-overlay-volume" pod "11a62bd2-9993-4188-9d00-54cfff26d2f0" (UID: "11a62bd2-9993-4188-9d00-54cfff26d2f0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:08.979573 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.979529 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-kube-api-access-xq2qq\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-kube-api-access-xq2qq\") pod \"11a62bd2-9993-4188-9d00-54cfff26d2f0\" (UID: \"11a62bd2-9993-4188-9d00-54cfff26d2f0\") " Oct 30 18:54:08.980228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.980206 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:08.980356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.980231 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a62bd2-9993-4188-9d00-54cfff26d2f0-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:08.981609 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.981585 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-provider-credentials" pod "11a62bd2-9993-4188-9d00-54cfff26d2f0" (UID: "11a62bd2-9993-4188-9d00-54cfff26d2f0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:08.981699 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:08.981639 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-kube-api-access-xq2qq" pod "11a62bd2-9993-4188-9d00-54cfff26d2f0" (UID: "11a62bd2-9993-4188-9d00-54cfff26d2f0"). InnerVolumeSpecName "kube-api-access-xq2qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:09.066194 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.066157 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31133e8f-0935-4d1c-9d0f-c253c3244094" path="/var/lib/kubelet/pods/31133e8f-0935-4d1c-9d0f-c253c3244094/volumes" Oct 30 18:54:09.081963 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.081942 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq2qq\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-kube-api-access-xq2qq\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:09.081963 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.081964 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11a62bd2-9993-4188-9d00-54cfff26d2f0-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:09.274062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.273975 4862 generic.go:356] "Generic (PLEG): container finished" podID="11a62bd2-9993-4188-9d00-54cfff26d2f0" containerID="871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05" exitCode=0 Oct 30 18:54:09.274062 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.274030 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" Oct 30 18:54:09.274615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.274071 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" event={"ID":"11a62bd2-9993-4188-9d00-54cfff26d2f0","Type":"ContainerDied","Data":"871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05"} Oct 30 18:54:09.274615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.274147 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn" event={"ID":"11a62bd2-9993-4188-9d00-54cfff26d2f0","Type":"ContainerDied","Data":"fd1170c397129c4641cd718c562986fc239a23e8c76ffa028892f3e5ba3f377b"} Oct 30 18:54:09.274615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.274187 4862 scope.go:122] "RemoveContainer" containerID="871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05" Oct 30 18:54:09.284957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.284936 4862 scope.go:122] "RemoveContainer" containerID="871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05" Oct 30 18:54:09.285216 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:54:09.285199 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05\": container with ID starting with 871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05 not found: ID does not exist" containerID="871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05" Oct 30 18:54:09.285332 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.285224 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05"} err="failed to get container status \"871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05\": rpc error: code = NotFound desc = could not find container \"871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05\": container with ID starting with 871e86f61576566b2601795eb500248dc857b696851251e88d1fe1ef450cad05 not found: ID does not exist" Oct 30 18:54:09.327539 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.327518 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn"] Oct 30 18:54:09.339547 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:09.339524 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2107d4/payload-processing-external-model-e2e-extmod-a-2107d4-9784rvwsn"] Oct 30 18:54:10.858757 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.858724 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs"] Oct 30 18:54:10.859526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.859505 4862 cpu_manager.go:414] "containerMap: removing container" podUID="11a62bd2-9993-4188-9d00-54cfff26d2f0" containerName="payload-processing" Oct 30 18:54:10.859526 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.859523 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a62bd2-9993-4188-9d00-54cfff26d2f0" containerName="payload-processing" Oct 30 18:54:10.859711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.859549 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a8f1b6ee-f340-428c-9287-f0f5630c3640" containerName="revoke-keys" Oct 30 18:54:10.859711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.859554 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f1b6ee-f340-428c-9287-f0f5630c3640" containerName="revoke-keys" Oct 30 18:54:10.859711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.859565 4862 cpu_manager.go:414] "containerMap: removing container" podUID="31133e8f-0935-4d1c-9d0f-c253c3244094" containerName="payload-processing" Oct 30 18:54:10.859711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.859570 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31133e8f-0935-4d1c-9d0f-c253c3244094" containerName="payload-processing" Oct 30 18:54:10.859711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.859673 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a62bd2-9993-4188-9d00-54cfff26d2f0" containerName="payload-processing" Oct 30 18:54:10.859711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.859682 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f1b6ee-f340-428c-9287-f0f5630c3640" containerName="revoke-keys" Oct 30 18:54:10.859711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.859688 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="31133e8f-0935-4d1c-9d0f-c253c3244094" containerName="payload-processing" Oct 30 18:54:10.864902 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.864880 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:10.896313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.882527 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2820d9-openshift-default-dockercfg-sftjd\"" Oct 30 18:54:10.903845 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.903816 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs"] Oct 30 18:54:10.926814 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.926773 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-data\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:10.927108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.926846 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istiod-ca-cert\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:10.927108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.926955 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdz5\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-kube-api-access-kbdz5\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:10.927108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.927004 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-credential-socket\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:10.927108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.927057 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-envoy\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:10.927108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.927101 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-podinfo\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:10.927401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.927126 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-certs\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:10.927401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.927171 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-token\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:10.927401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:10.927278 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-socket\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.029370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.029335 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-socket\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.029763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.029735 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-data\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.029924 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.029906 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istiod-ca-cert\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.030060 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.030045 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdz5\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-kube-api-access-kbdz5\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.030192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.030178 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-credential-socket\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.030342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.030326 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-envoy\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.030476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.030459 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-podinfo\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.030600 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.030584 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-certs\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.030747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.030732 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-token\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.036077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.032320 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-credential-socket\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.036077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.034659 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-data\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.036077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.035147 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istiod-ca-cert\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.036077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.035473 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-certs\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.036077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.035915 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-socket\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.128334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.109859 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-token\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.128334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.122102 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdz5\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-kube-api-access-kbdz5\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.128334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.124633 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-envoy\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.155288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.152126 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-podinfo\") pod \"e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.171282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.164875 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a62bd2-9993-4188-9d00-54cfff26d2f0" path="/var/lib/kubelet/pods/11a62bd2-9993-4188-9d00-54cfff26d2f0/volumes" Oct 30 18:54:11.182102 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.180580 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:11.593494 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:54:11.593458 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c42fe79_10c6_4985_8d9e_acdd579ebcca.slice/crio-7b8076362fa3f35dd0497f299eebbf75b3255765433e9bce22ac2587536a3b75 WatchSource:0}: Error finding container 7b8076362fa3f35dd0497f299eebbf75b3255765433e9bce22ac2587536a3b75: Status 404 returned error can't find the container with id 7b8076362fa3f35dd0497f299eebbf75b3255765433e9bce22ac2587536a3b75 Oct 30 18:54:11.593851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.593830 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs"] Oct 30 18:54:11.596382 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.596352 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:11.596481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.596414 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:11.596481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:11.596444 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:12.311071 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:12.311038 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" event={"ID":"0c42fe79-10c6-4985-8d9e-acdd579ebcca","Type":"ContainerStarted","Data":"496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e"} Oct 30 18:54:12.311587 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:12.311094 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" event={"ID":"0c42fe79-10c6-4985-8d9e-acdd579ebcca","Type":"ContainerStarted","Data":"7b8076362fa3f35dd0497f299eebbf75b3255765433e9bce22ac2587536a3b75"} Oct 30 18:54:13.181438 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:13.181406 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:13.187907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:13.187875 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:13.246535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:13.246486 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" podStartSLOduration=3.246473203 podStartE2EDuration="3.246473203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:12.386310577 +0000 UTC m=+3351.435355491" watchObservedRunningTime="2026-10-30 18:54:13.246473203 +0000 UTC m=+3352.295518115" Oct 30 18:54:13.319403 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:13.319378 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:13.320515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:13.320495 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:54:15.292629 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:15.292601 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 18:54:16.470844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.470809 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk"] Oct 30 18:54:16.478189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.478160 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.502688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.502337 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6018b4\"" Oct 30 18:54:16.502688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.502449 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"routing-overlay\"" Oct 30 18:54:16.502688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.502512 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"agc-cred-8403349d486c\"" Oct 30 18:54:16.502688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.502560 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"agc-cred-45d23ae5f8fc\"" Oct 30 18:54:16.502688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.502635 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"kube-root-ca.crt\"" Oct 30 18:54:16.514813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.511451 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"openshift-service-ca.crt\"" Oct 30 18:54:16.514813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.511554 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6018b4\"/\"payload-processing-external-model-e2e--6d8a874f-dockercfg-j6h5f\"" Oct 30 18:54:16.526360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.526332 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk"] Oct 30 18:54:16.553869 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.553833 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.554055 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.554034 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.554115 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.554078 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j2sc\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-kube-api-access-9j2sc\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.554384 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.554359 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.655789 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.655755 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.655789 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.655792 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j2sc\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-kube-api-access-9j2sc\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.656010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.655895 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.656010 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.655963 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.656632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.656605 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.656772 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.656688 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.658493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.658467 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.675758 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.675736 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j2sc\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-kube-api-access-9j2sc\") pod \"payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:16.805076 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:16.805054 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:17.211310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.211283 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk"] Oct 30 18:54:17.375851 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.375815 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj"] Oct 30 18:54:17.467052 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.466970 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" event={"ID":"6eae747d-2b0a-448e-bd7c-5e98194f3e52","Type":"ContainerStarted","Data":"12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897"} Oct 30 18:54:17.467293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.467251 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" event={"ID":"6eae747d-2b0a-448e-bd7c-5e98194f3e52","Type":"ContainerStarted","Data":"718dc41eba1c6b0f860999f2acb18ab2efcdf34ff2048d9ec070a258864492ce"} Oct 30 18:54:17.467437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.467426 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:17.467534 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.467524 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj"] Oct 30 18:54:17.467625 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.467107 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.476016 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.475961 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" podStartSLOduration=1.475944307 podStartE2EDuration="1.475944307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:17.471125173 +0000 UTC m=+3356.520170086" watchObservedRunningTime="2026-10-30 18:54:17.475944307 +0000 UTC m=+3356.524989221" Oct 30 18:54:17.485045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.484482 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:54:17.485045 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.484582 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:54:17.520108 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.520072 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b673134-kgfbj\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.520299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.520149 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b673134-kgfbj\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.520515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.520492 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrptw\" (UniqueName: \"kubernetes.io/projected/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-kube-api-access-hrptw\") pod \"maas-api-revoke-sub-5b673134-kgfbj\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.622647 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.622553 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrptw\" (UniqueName: \"kubernetes.io/projected/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-kube-api-access-hrptw\") pod \"maas-api-revoke-sub-5b673134-kgfbj\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.622891 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.622847 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b673134-kgfbj\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.622988 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.622898 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b673134-kgfbj\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.623636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.623614 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b673134-kgfbj\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.626221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.626195 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b673134-kgfbj\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.652095 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.652061 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrptw\" (UniqueName: \"kubernetes.io/projected/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-kube-api-access-hrptw\") pod \"maas-api-revoke-sub-5b673134-kgfbj\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:17.795500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:17.795477 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:18.102539 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:54:18.102507 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5c2a4c_ad82_4dd3_9e5e_820664cc443f.slice/crio-ff05153578d5f3ace733b533d6892f91d205f3f082226ed53f0b4c5733a0dcf8 WatchSource:0}: Error finding container ff05153578d5f3ace733b533d6892f91d205f3f082226ed53f0b4c5733a0dcf8: Status 404 returned error can't find the container with id ff05153578d5f3ace733b533d6892f91d205f3f082226ed53f0b4c5733a0dcf8 Oct 30 18:54:18.102854 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.102835 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj"] Oct 30 18:54:18.409696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.409617 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" event={"ID":"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f","Type":"ContainerStarted","Data":"ff05153578d5f3ace733b533d6892f91d205f3f082226ed53f0b4c5733a0dcf8"} Oct 30 18:54:18.523100 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.523072 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq"] Oct 30 18:54:18.529149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.529131 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:18.545595 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.545570 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2820d9\"" Oct 30 18:54:18.545725 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.545701 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq"] Oct 30 18:54:18.552584 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.552557 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2820d9-dockercfg-hvf7j\"" Oct 30 18:54:18.724340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.723186 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34f6da0-5ece-4140-8889-e6bea714f874-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq\" (UID: \"d34f6da0-5ece-4140-8889-e6bea714f874\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:18.798289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.795084 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6b4z\" (UniqueName: \"kubernetes.io/projected/d34f6da0-5ece-4140-8889-e6bea714f874-kube-api-access-f6b4z\") pod \"payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq\" (UID: \"d34f6da0-5ece-4140-8889-e6bea714f874\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:18.806329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.804310 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p"] Oct 30 18:54:18.814376 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.809716 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:18.821106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.821055 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p"] Oct 30 18:54:18.896628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.896594 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxwl\" (UniqueName: \"kubernetes.io/projected/ab801c39-eeb3-40da-8353-2d7c9dab9181-kube-api-access-9qxwl\") pod \"payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p\" (UID: \"ab801c39-eeb3-40da-8353-2d7c9dab9181\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:18.896844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.896736 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab801c39-eeb3-40da-8353-2d7c9dab9181-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p\" (UID: \"ab801c39-eeb3-40da-8353-2d7c9dab9181\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:18.896971 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.896952 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34f6da0-5ece-4140-8889-e6bea714f874-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq\" (UID: \"d34f6da0-5ece-4140-8889-e6bea714f874\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:18.897053 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.897037 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6b4z\" (UniqueName: \"kubernetes.io/projected/d34f6da0-5ece-4140-8889-e6bea714f874-kube-api-access-f6b4z\") pod \"payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq\" (UID: \"d34f6da0-5ece-4140-8889-e6bea714f874\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:18.897661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.897643 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34f6da0-5ece-4140-8889-e6bea714f874-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq\" (UID: \"d34f6da0-5ece-4140-8889-e6bea714f874\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:18.916510 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.916487 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6b4z\" (UniqueName: \"kubernetes.io/projected/d34f6da0-5ece-4140-8889-e6bea714f874-kube-api-access-f6b4z\") pod \"payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq\" (UID: \"d34f6da0-5ece-4140-8889-e6bea714f874\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:18.998809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.998739 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxwl\" (UniqueName: \"kubernetes.io/projected/ab801c39-eeb3-40da-8353-2d7c9dab9181-kube-api-access-9qxwl\") pod \"payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p\" (UID: \"ab801c39-eeb3-40da-8353-2d7c9dab9181\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:18.999009 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.998990 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab801c39-eeb3-40da-8353-2d7c9dab9181-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p\" (UID: \"ab801c39-eeb3-40da-8353-2d7c9dab9181\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:18.999753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:18.999731 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab801c39-eeb3-40da-8353-2d7c9dab9181-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p\" (UID: \"ab801c39-eeb3-40da-8353-2d7c9dab9181\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:19.020820 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.020795 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxwl\" (UniqueName: \"kubernetes.io/projected/ab801c39-eeb3-40da-8353-2d7c9dab9181-kube-api-access-9qxwl\") pod \"payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p\" (UID: \"ab801c39-eeb3-40da-8353-2d7c9dab9181\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:19.030106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.030078 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd"] Oct 30 18:54:19.034648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.034627 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8d63a5-72dqd"] Oct 30 18:54:19.065611 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.065588 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c6d33c-5374-4ad0-b00d-60c1ba64dec7" path="/var/lib/kubelet/pods/09c6d33c-5374-4ad0-b00d-60c1ba64dec7/volumes" Oct 30 18:54:19.149521 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.149490 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:19.150576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.150557 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:19.428523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.427734 4862 generic.go:356] "Generic (PLEG): container finished" podID="0c5c2a4c-ad82-4dd3-9e5e-820664cc443f" containerID="2823c4093e1e433eaac651107a0bed1878856fd486c8823b4597594e205b1dc2" exitCode=0 Oct 30 18:54:19.429427 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.429390 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" event={"ID":"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f","Type":"ContainerDied","Data":"2823c4093e1e433eaac651107a0bed1878856fd486c8823b4597594e205b1dc2"} Oct 30 18:54:19.472127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.472095 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq"] Oct 30 18:54:19.472257 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:54:19.472164 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34f6da0_5ece_4140_8889_e6bea714f874.slice/crio-db3f9c0f33253a29000130eb01ff839c6493587d0be2d14d25ba5bb670e4d4da WatchSource:0}: Error finding container db3f9c0f33253a29000130eb01ff839c6493587d0be2d14d25ba5bb670e4d4da: Status 404 returned error can't find the container with id db3f9c0f33253a29000130eb01ff839c6493587d0be2d14d25ba5bb670e4d4da Oct 30 18:54:19.485753 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:19.485732 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p"] Oct 30 18:54:19.488771 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:54:19.488746 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab801c39_eeb3_40da_8353_2d7c9dab9181.slice/crio-a5b0620d8bf8f816ddd2ef9d7e991c2a429987eafb58410bc1faf440734e9dfd WatchSource:0}: Error finding container a5b0620d8bf8f816ddd2ef9d7e991c2a429987eafb58410bc1faf440734e9dfd: Status 404 returned error can't find the container with id a5b0620d8bf8f816ddd2ef9d7e991c2a429987eafb58410bc1faf440734e9dfd Oct 30 18:54:19.874334 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:54:19.874298 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:54:20.438281 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.438235 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" event={"ID":"d34f6da0-5ece-4140-8889-e6bea714f874","Type":"ContainerStarted","Data":"deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d"} Oct 30 18:54:20.438831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.438808 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:20.438957 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.438863 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" event={"ID":"d34f6da0-5ece-4140-8889-e6bea714f874","Type":"ContainerStarted","Data":"db3f9c0f33253a29000130eb01ff839c6493587d0be2d14d25ba5bb670e4d4da"} Oct 30 18:54:20.440329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.440306 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" event={"ID":"ab801c39-eeb3-40da-8353-2d7c9dab9181","Type":"ContainerStarted","Data":"1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c"} Oct 30 18:54:20.440440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.440359 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:20.440440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.440378 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" event={"ID":"ab801c39-eeb3-40da-8353-2d7c9dab9181","Type":"ContainerStarted","Data":"a5b0620d8bf8f816ddd2ef9d7e991c2a429987eafb58410bc1faf440734e9dfd"} Oct 30 18:54:20.480351 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.480290 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" podStartSLOduration=2.480252598 podStartE2EDuration="2.480252598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:20.477326691 +0000 UTC m=+3359.526371605" watchObservedRunningTime="2026-10-30 18:54:20.480252598 +0000 UTC m=+3359.529297512" Oct 30 18:54:20.534569 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.534515 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" podStartSLOduration=2.53449722 podStartE2EDuration="2.53449722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:20.520341372 +0000 UTC m=+3359.569386288" watchObservedRunningTime="2026-10-30 18:54:20.53449722 +0000 UTC m=+3359.583542134" Oct 30 18:54:20.621922 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.621895 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:20.728859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.728765 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-xks-ca\") pod \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " Oct 30 18:54:20.728859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.728833 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-kube-api-access-hrptw\" (UniqueName: \"kubernetes.io/projected/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-kube-api-access-hrptw\") pod \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " Oct 30 18:54:20.729104 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.728865 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-openshift-ca\") pod \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\" (UID: \"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f\") " Oct 30 18:54:20.729604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.729577 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-openshift-ca" pod "0c5c2a4c-ad82-4dd3-9e5e-820664cc443f" (UID: "0c5c2a4c-ad82-4dd3-9e5e-820664cc443f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:20.739688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.739664 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-xks-ca" pod "0c5c2a4c-ad82-4dd3-9e5e-820664cc443f" (UID: "0c5c2a4c-ad82-4dd3-9e5e-820664cc443f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:20.741656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.741630 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-kube-api-access-hrptw" pod "0c5c2a4c-ad82-4dd3-9e5e-820664cc443f" (UID: "0c5c2a4c-ad82-4dd3-9e5e-820664cc443f"). InnerVolumeSpecName "kube-api-access-hrptw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:20.831366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.831338 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:20.831366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.831364 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrptw\" (UniqueName: \"kubernetes.io/projected/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-kube-api-access-hrptw\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:20.831530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:20.831376 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:21.449942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:21.449863 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" Oct 30 18:54:21.450451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:21.449970 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj" event={"ID":"0c5c2a4c-ad82-4dd3-9e5e-820664cc443f","Type":"ContainerDied","Data":"ff05153578d5f3ace733b533d6892f91d205f3f082226ed53f0b4c5733a0dcf8"} Oct 30 18:54:21.450451 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:21.450024 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff05153578d5f3ace733b533d6892f91d205f3f082226ed53f0b4c5733a0dcf8" Oct 30 18:54:22.492366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:22.492338 4862 scope.go:122] "RemoveContainer" containerID="f4418affcce0430ff6c674b2145b6ebda572b89559ea644fa9eb7cc41e4a61ac" Oct 30 18:54:22.503439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:22.503413 4862 scope.go:122] "RemoveContainer" containerID="6b7594d8f5d02cbfca0d1507985c359927e3196c9faba469b34bbccfc623f214" Oct 30 18:54:28.029896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.029855 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg"] Oct 30 18:54:28.041837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.041811 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-347a9bb0-m27tg"] Oct 30 18:54:28.450275 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.450235 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn"] Oct 30 18:54:28.451006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.450990 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0c5c2a4c-ad82-4dd3-9e5e-820664cc443f" containerName="revoke-keys" Oct 30 18:54:28.451058 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.451008 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5c2a4c-ad82-4dd3-9e5e-820664cc443f" containerName="revoke-keys" Oct 30 18:54:28.451138 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.451129 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5c2a4c-ad82-4dd3-9e5e-820664cc443f" containerName="revoke-keys" Oct 30 18:54:28.456672 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.456651 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.471355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.471330 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:54:28.473321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.473298 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn"] Oct 30 18:54:28.480543 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.480518 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:54:28.549535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.549508 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbrw\" (UniqueName: \"kubernetes.io/projected/b8222f63-efc7-419b-afff-bb79fe5a599e-kube-api-access-tcbrw\") pod \"maas-api-revoke-sub-5622de06-ppjqn\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.549676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.549620 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5622de06-ppjqn\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.549762 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.549747 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5622de06-ppjqn\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.651437 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.651409 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbrw\" (UniqueName: \"kubernetes.io/projected/b8222f63-efc7-419b-afff-bb79fe5a599e-kube-api-access-tcbrw\") pod \"maas-api-revoke-sub-5622de06-ppjqn\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.651585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.651466 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5622de06-ppjqn\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.651585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.651548 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5622de06-ppjqn\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.652145 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.652120 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5622de06-ppjqn\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.654075 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.654056 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5622de06-ppjqn\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.670892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.670871 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbrw\" (UniqueName: \"kubernetes.io/projected/b8222f63-efc7-419b-afff-bb79fe5a599e-kube-api-access-tcbrw\") pod \"maas-api-revoke-sub-5622de06-ppjqn\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:28.772483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:28.772423 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:29.064473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:29.064446 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b6e9fb-aeaf-4661-8325-8b15cbfef68d" path="/var/lib/kubelet/pods/92b6e9fb-aeaf-4661-8325-8b15cbfef68d/volumes" Oct 30 18:54:29.081138 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:54:29.081109 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8222f63_efc7_419b_afff_bb79fe5a599e.slice/crio-a3340f2d8ae26d7da247b3ee60b5fdf41a4577cef95c26bb9bc94a7518c644d9 WatchSource:0}: Error finding container a3340f2d8ae26d7da247b3ee60b5fdf41a4577cef95c26bb9bc94a7518c644d9: Status 404 returned error can't find the container with id a3340f2d8ae26d7da247b3ee60b5fdf41a4577cef95c26bb9bc94a7518c644d9 Oct 30 18:54:29.081626 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:29.081604 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn"] Oct 30 18:54:29.432401 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:29.432375 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:29.517980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:29.517946 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" event={"ID":"b8222f63-efc7-419b-afff-bb79fe5a599e","Type":"ContainerStarted","Data":"4379a7548d64d079d6b1e79dcae1c0e7cb3e0965e94bdf62df12c08547456c07"} Oct 30 18:54:29.518149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:29.518011 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" event={"ID":"b8222f63-efc7-419b-afff-bb79fe5a599e","Type":"ContainerStarted","Data":"a3340f2d8ae26d7da247b3ee60b5fdf41a4577cef95c26bb9bc94a7518c644d9"} Oct 30 18:54:29.558576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:29.558530 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" podStartSLOduration=1.558512029 podStartE2EDuration="1.558512029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:29.555872509 +0000 UTC m=+3368.604917421" watchObservedRunningTime="2026-10-30 18:54:29.558512029 +0000 UTC m=+3368.607557005" Oct 30 18:54:30.527979 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:30.527948 4862 generic.go:356] "Generic (PLEG): container finished" podID="b8222f63-efc7-419b-afff-bb79fe5a599e" containerID="4379a7548d64d079d6b1e79dcae1c0e7cb3e0965e94bdf62df12c08547456c07" exitCode=0 Oct 30 18:54:30.528397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:30.528031 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" event={"ID":"b8222f63-efc7-419b-afff-bb79fe5a599e","Type":"ContainerDied","Data":"4379a7548d64d079d6b1e79dcae1c0e7cb3e0965e94bdf62df12c08547456c07"} Oct 30 18:54:31.454773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.454701 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 18:54:31.713331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.713228 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:31.819921 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.810046 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8222f63-efc7-419b-afff-bb79fe5a599e-kube-api-access-tcbrw\" (UniqueName: \"kubernetes.io/projected/b8222f63-efc7-419b-afff-bb79fe5a599e-kube-api-access-tcbrw\") pod \"b8222f63-efc7-419b-afff-bb79fe5a599e\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " Oct 30 18:54:31.819921 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.810244 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-openshift-ca\") pod \"b8222f63-efc7-419b-afff-bb79fe5a599e\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " Oct 30 18:54:31.819921 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.810441 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-xks-ca\") pod \"b8222f63-efc7-419b-afff-bb79fe5a599e\" (UID: \"b8222f63-efc7-419b-afff-bb79fe5a599e\") " Oct 30 18:54:31.840411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.840366 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-openshift-ca" pod "b8222f63-efc7-419b-afff-bb79fe5a599e" (UID: "b8222f63-efc7-419b-afff-bb79fe5a599e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:31.841038 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.840978 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-xks-ca" pod "b8222f63-efc7-419b-afff-bb79fe5a599e" (UID: "b8222f63-efc7-419b-afff-bb79fe5a599e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:54:31.844213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.844187 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8222f63-efc7-419b-afff-bb79fe5a599e-kube-api-access-tcbrw" pod "b8222f63-efc7-419b-afff-bb79fe5a599e" (UID: "b8222f63-efc7-419b-afff-bb79fe5a599e"). InnerVolumeSpecName "kube-api-access-tcbrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:31.912667 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.912637 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:31.912667 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.912666 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcbrw\" (UniqueName: \"kubernetes.io/projected/b8222f63-efc7-419b-afff-bb79fe5a599e-kube-api-access-tcbrw\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:31.912846 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:31.912677 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8222f63-efc7-419b-afff-bb79fe5a599e-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:32.461000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:32.460974 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 18:54:32.549324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:32.549295 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" Oct 30 18:54:32.549497 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:32.549291 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn" event={"ID":"b8222f63-efc7-419b-afff-bb79fe5a599e","Type":"ContainerDied","Data":"a3340f2d8ae26d7da247b3ee60b5fdf41a4577cef95c26bb9bc94a7518c644d9"} Oct 30 18:54:32.549497 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:32.549402 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3340f2d8ae26d7da247b3ee60b5fdf41a4577cef95c26bb9bc94a7518c644d9" Oct 30 18:54:43.079731 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:43.079698 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk"] Oct 30 18:54:48.080013 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.079972 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" podUID="6eae747d-2b0a-448e-bd7c-5e98194f3e52" containerName="payload-processing" containerID="cri-o://12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897" gracePeriod=25 Oct 30 18:54:48.389378 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.389340 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:48.495508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.495473 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-provider-credentials\") pod \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " Oct 30 18:54:48.495706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.495514 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-kube-api-access-9j2sc\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-kube-api-access-9j2sc\") pod \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " Oct 30 18:54:48.495706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.495557 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-routing-overlay-volume\") pod \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " Oct 30 18:54:48.495706 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.495692 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-plugins-config-volume\") pod \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\" (UID: \"6eae747d-2b0a-448e-bd7c-5e98194f3e52\") " Oct 30 18:54:48.496197 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.496107 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-routing-overlay-volume" pod "6eae747d-2b0a-448e-bd7c-5e98194f3e52" (UID: "6eae747d-2b0a-448e-bd7c-5e98194f3e52"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:48.496395 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.496357 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-plugins-config-volume" pod "6eae747d-2b0a-448e-bd7c-5e98194f3e52" (UID: "6eae747d-2b0a-448e-bd7c-5e98194f3e52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:54:48.496612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.496590 4862 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-routing-overlay-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:48.496708 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.496620 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eae747d-2b0a-448e-bd7c-5e98194f3e52-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:48.497942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.497910 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-provider-credentials" pod "6eae747d-2b0a-448e-bd7c-5e98194f3e52" (UID: "6eae747d-2b0a-448e-bd7c-5e98194f3e52"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:48.498051 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.497992 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-kube-api-access-9j2sc" pod "6eae747d-2b0a-448e-bd7c-5e98194f3e52" (UID: "6eae747d-2b0a-448e-bd7c-5e98194f3e52"). InnerVolumeSpecName "kube-api-access-9j2sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:54:48.598803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.598719 4862 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-provider-credentials\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:48.598803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.598755 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j2sc\" (UniqueName: \"kubernetes.io/projected/6eae747d-2b0a-448e-bd7c-5e98194f3e52-kube-api-access-9j2sc\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:54:48.828313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.828277 4862 generic.go:356] "Generic (PLEG): container finished" podID="6eae747d-2b0a-448e-bd7c-5e98194f3e52" containerID="12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897" exitCode=0 Oct 30 18:54:48.828313 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.828294 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" event={"ID":"6eae747d-2b0a-448e-bd7c-5e98194f3e52","Type":"ContainerDied","Data":"12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897"} Oct 30 18:54:48.828542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.828362 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" event={"ID":"6eae747d-2b0a-448e-bd7c-5e98194f3e52","Type":"ContainerDied","Data":"718dc41eba1c6b0f860999f2acb18ab2efcdf34ff2048d9ec070a258864492ce"} Oct 30 18:54:48.828542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.828363 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk" Oct 30 18:54:48.828542 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.828391 4862 scope.go:122] "RemoveContainer" containerID="12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897" Oct 30 18:54:48.841385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.841363 4862 scope.go:122] "RemoveContainer" containerID="12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897" Oct 30 18:54:48.841693 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:54:48.841674 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897\": container with ID starting with 12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897 not found: ID does not exist" containerID="12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897" Oct 30 18:54:48.841748 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.841702 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897"} err="failed to get container status \"12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897\": rpc error: code = NotFound desc = could not find container \"12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897\": container with ID starting with 12abe742fca5df50b9230a4aad61639cb18a50a4acf50e571fc0d14ed68bf897 not found: ID does not exist" Oct 30 18:54:48.883032 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.882962 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk"] Oct 30 18:54:48.886943 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:48.886919 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6018b4/payload-processing-external-model-e2e-extmod-b-6018b4-6dc5qbjfk"] Oct 30 18:54:49.065279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:49.065240 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eae747d-2b0a-448e-bd7c-5e98194f3e52" path="/var/lib/kubelet/pods/6eae747d-2b0a-448e-bd7c-5e98194f3e52/volumes" Oct 30 18:54:49.872844 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:54:49.872803 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:54:53.164355 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.164322 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr"] Oct 30 18:54:53.165085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.165063 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6eae747d-2b0a-448e-bd7c-5e98194f3e52" containerName="payload-processing" Oct 30 18:54:53.165085 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.165082 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eae747d-2b0a-448e-bd7c-5e98194f3e52" containerName="payload-processing" Oct 30 18:54:53.165307 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.165151 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b8222f63-efc7-419b-afff-bb79fe5a599e" containerName="revoke-keys" Oct 30 18:54:53.165307 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.165157 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8222f63-efc7-419b-afff-bb79fe5a599e" containerName="revoke-keys" Oct 30 18:54:53.165307 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.165306 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8222f63-efc7-419b-afff-bb79fe5a599e" containerName="revoke-keys" Oct 30 18:54:53.165461 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.165320 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eae747d-2b0a-448e-bd7c-5e98194f3e52" containerName="payload-processing" Oct 30 18:54:53.169890 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.169870 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.194511 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.194488 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr"] Oct 30 18:54:53.200556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.200534 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-861bb3-openshift-default-dockercfg-vns5c\"" Oct 30 18:54:53.205189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.205164 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9"] Oct 30 18:54:53.209552 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.209533 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.228502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.228477 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9"] Oct 30 18:54:53.266159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.266127 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkprx\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-kube-api-access-pkprx\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.266341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.266306 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.266341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.266337 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-credential-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.266440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.266365 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-certs\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.266481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.266460 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1db3a62-9493-4e08-ad00-6099feb138df-istiod-ca-cert\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.266562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.266548 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-envoy\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.266643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.266631 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1db3a62-9493-4e08-ad00-6099feb138df-istio-podinfo\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.266705 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.266685 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-data\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.266927 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.266847 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-istio-token\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.367897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.367867 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1db3a62-9493-4e08-ad00-6099feb138df-istio-podinfo\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.368034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.367920 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.368034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.367939 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-data\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.368034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.367967 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97g98\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-kube-api-access-97g98\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.368143 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368117 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-istio-token\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.368192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368163 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.368246 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368216 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.368325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368245 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558226ee-2091-4c41-a0e8-70591a91a3cd-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.368325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368308 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-data\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.368439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368409 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkprx\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-kube-api-access-pkprx\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.368560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368543 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.368627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368580 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-credential-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.368627 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368614 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.368737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368647 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-certs\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.368737 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368718 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1db3a62-9493-4e08-ad00-6099feb138df-istiod-ca-cert\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.368850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368750 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.368850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368799 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.368850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368832 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.369024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368848 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.369024 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.368869 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-envoy\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.369133 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.369064 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-certs\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.369174 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.369144 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-credential-socket\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.369439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.369415 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1db3a62-9493-4e08-ad00-6099feb138df-istiod-ca-cert\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.370597 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.370579 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1db3a62-9493-4e08-ad00-6099feb138df-istio-podinfo\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.371144 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.371117 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-envoy\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.387252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.387230 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkprx\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-kube-api-access-pkprx\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.387502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.387481 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-istio-token\") pod \"e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.470301 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470242 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97g98\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-kube-api-access-97g98\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470317 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470349 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470387 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470366 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558226ee-2091-4c41-a0e8-70591a91a3cd-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470472 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470512 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470502 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470535 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470565 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470743 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470646 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470802 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470739 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-certs\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470839 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470884 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-data\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.470962 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.470941 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-credential-socket\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.471111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.471091 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558226ee-2091-4c41-a0e8-70591a91a3cd-istiod-ca-cert\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.472717 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.472699 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-envoy\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.473276 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.473248 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-podinfo\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.484822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.484800 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:53.499198 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.499175 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-token\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.499423 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.499404 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97g98\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-kube-api-access-97g98\") pod \"e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.532107 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.532077 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:53.815374 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:54:53.815346 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1db3a62_9493_4e08_ad00_6099feb138df.slice/crio-4161aa7fae8603d557c1b6810613d213b35304074eee221f43064f8950e3b283 WatchSource:0}: Error finding container 4161aa7fae8603d557c1b6810613d213b35304074eee221f43064f8950e3b283: Status 404 returned error can't find the container with id 4161aa7fae8603d557c1b6810613d213b35304074eee221f43064f8950e3b283 Oct 30 18:54:53.816590 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.816562 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr"] Oct 30 18:54:53.818408 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.818379 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:53.818476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.818446 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:53.818522 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.818475 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:53.877440 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.877411 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" event={"ID":"f1db3a62-9493-4e08-ad00-6099feb138df","Type":"ContainerStarted","Data":"4161aa7fae8603d557c1b6810613d213b35304074eee221f43064f8950e3b283"} Oct 30 18:54:53.885701 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:54:53.885654 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558226ee_2091_4c41_a0e8_70591a91a3cd.slice/crio-e91950b6fe9af3482aaec300498cc2fb5bd4a8fad8a246073f553591845dcbba WatchSource:0}: Error finding container e91950b6fe9af3482aaec300498cc2fb5bd4a8fad8a246073f553591845dcbba: Status 404 returned error can't find the container with id e91950b6fe9af3482aaec300498cc2fb5bd4a8fad8a246073f553591845dcbba Oct 30 18:54:53.886533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.886398 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9"] Oct 30 18:54:53.888747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.888720 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:53.888820 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.888784 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:53.888859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:53.888817 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:54:54.897799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:54.897762 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" event={"ID":"558226ee-2091-4c41-a0e8-70591a91a3cd","Type":"ContainerStarted","Data":"44b17779e9df9ec48e1d486c82e3436f1b8a05105ff53cb7935ae16275dfaebe"} Oct 30 18:54:54.898210 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:54.897829 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" event={"ID":"558226ee-2091-4c41-a0e8-70591a91a3cd","Type":"ContainerStarted","Data":"e91950b6fe9af3482aaec300498cc2fb5bd4a8fad8a246073f553591845dcbba"} Oct 30 18:54:54.899460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:54.899439 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" event={"ID":"f1db3a62-9493-4e08-ad00-6099feb138df","Type":"ContainerStarted","Data":"2ea677fa962853f52cd7fc857a9fc15b3bfc22a2b3397b922ff996bf159c4c4e"} Oct 30 18:54:55.011643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:55.011587 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" podStartSLOduration=2.011574023 podStartE2EDuration="2.011574023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:54.96371876 +0000 UTC m=+3394.012763675" watchObservedRunningTime="2026-10-30 18:54:55.011574023 +0000 UTC m=+3394.060618935" Oct 30 18:54:55.012983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:55.012951 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" podStartSLOduration=2.012940473 podStartE2EDuration="2.012940473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:55.009455753 +0000 UTC m=+3394.058500667" watchObservedRunningTime="2026-10-30 18:54:55.012940473 +0000 UTC m=+3394.061985386" Oct 30 18:54:55.486549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:55.485590 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:54:55.487687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:55.486877 4862 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 18:54:55.487687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:55.486924 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 18:54:55.558959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:55.551627 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:54:55.567299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:55.559370 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:54:55.567299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:55.559432 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 18:54:56.486803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:56.486373 4862 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 18:54:56.486803 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:56.486438 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 18:54:56.535605 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:56.535570 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:54:56.535868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:56.535842 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 18:54:57.486530 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:57.486481 4862 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 18:54:57.486730 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:57.486554 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 18:54:57.561286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:57.560787 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:54:57.561286 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:57.560847 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 18:54:58.488688 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:58.488643 4862 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 18:54:58.488905 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:58.488712 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 18:54:58.543852 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:58.543821 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:54:58.544160 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:58.544115 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 18:54:59.485249 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:59.485215 4862 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 18:54:59.485663 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:59.485283 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 18:54:59.536394 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:59.536359 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:54:59.536677 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:54:59.536646 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 18:55:00.486284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:00.485686 4862 patch_prober.go:29] interesting pod/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 18:55:00.486284 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:00.485748 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 18:55:00.537289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:00.536553 4862 patch_prober.go:29] interesting pod/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 18:55:00.537289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:00.536610 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 18:55:01.491086 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:01.491056 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:55:01.491630 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:01.491609 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:55:01.492367 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:01.492343 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 18:55:01.611422 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:01.611392 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:55:01.613397 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:01.613373 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:55:01.625504 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:01.625480 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 18:55:05.928248 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:05.928216 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v"] Oct 30 18:55:05.971905 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:05.971876 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:06.012124 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.012047 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-869b55-dockercfg-llldk\"" Oct 30 18:55:06.012432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.012412 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-869b55\"" Oct 30 18:55:06.017747 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.017719 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v"] Oct 30 18:55:06.070906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.070869 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0ab5dd-725d-4c8c-9530-311383a99f96-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v\" (UID: \"2b0ab5dd-725d-4c8c-9530-311383a99f96\") " pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:06.071176 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.071154 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5tj\" (UniqueName: \"kubernetes.io/projected/2b0ab5dd-725d-4c8c-9530-311383a99f96-kube-api-access-7f5tj\") pod \"payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v\" (UID: \"2b0ab5dd-725d-4c8c-9530-311383a99f96\") " pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:06.172993 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.172956 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0ab5dd-725d-4c8c-9530-311383a99f96-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v\" (UID: \"2b0ab5dd-725d-4c8c-9530-311383a99f96\") " pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:06.173182 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.173006 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5tj\" (UniqueName: \"kubernetes.io/projected/2b0ab5dd-725d-4c8c-9530-311383a99f96-kube-api-access-7f5tj\") pod \"payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v\" (UID: \"2b0ab5dd-725d-4c8c-9530-311383a99f96\") " pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:06.173934 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.173908 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0ab5dd-725d-4c8c-9530-311383a99f96-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v\" (UID: \"2b0ab5dd-725d-4c8c-9530-311383a99f96\") " pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:06.194324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.194241 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5tj\" (UniqueName: \"kubernetes.io/projected/2b0ab5dd-725d-4c8c-9530-311383a99f96-kube-api-access-7f5tj\") pod \"payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v\" (UID: \"2b0ab5dd-725d-4c8c-9530-311383a99f96\") " pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:06.335420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.335388 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:06.671559 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:06.667766 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v"] Oct 30 18:55:07.087448 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:07.087418 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" event={"ID":"2b0ab5dd-725d-4c8c-9530-311383a99f96","Type":"ContainerStarted","Data":"ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad"} Oct 30 18:55:07.087869 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:07.087476 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" event={"ID":"2b0ab5dd-725d-4c8c-9530-311383a99f96","Type":"ContainerStarted","Data":"84d769d563b247fc576872200f3bb18b4eff070215690d7fefc9de1389a86e79"} Oct 30 18:55:07.087869 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:07.087505 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:07.130044 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:07.129996 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" podStartSLOduration=2.129982504 podStartE2EDuration="2.129982504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:07.125048977 +0000 UTC m=+3406.174093889" watchObservedRunningTime="2026-10-30 18:55:07.129982504 +0000 UTC m=+3406.179027416" Oct 30 18:55:18.098975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:18.098945 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 18:55:19.901419 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:55:19.901376 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:55:22.563159 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:22.563127 4862 scope.go:122] "RemoveContainer" containerID="ada9954f67e0a585a75107333af0b121ea37ac2f1896f65fb278d6384b0ba511" Oct 30 18:55:22.582097 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:22.582069 4862 scope.go:122] "RemoveContainer" containerID="96caa73dbb63cbdedfe8ea04707f9f8ab636cbac1527c303b6876d166766ef5a" Oct 30 18:55:25.257602 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:25.257569 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:55:40.219704 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.219671 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l"] Oct 30 18:55:40.224821 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.224800 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.247806 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.247772 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l"] Oct 30 18:55:40.253114 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.253074 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f37ca7\"/\"kube-root-ca.crt\"" Oct 30 18:55:40.253114 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.253090 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f37ca7\"/\"test-model-f37ca7-kserve-self-signed-certs\"" Oct 30 18:55:40.253318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.253104 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f37ca7\"/\"openshift-service-ca.crt\"" Oct 30 18:55:40.253318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.253223 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f37ca7\"/\"default-dockercfg-lq9wm\"" Oct 30 18:55:40.282090 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.282060 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tls-certs\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.282230 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.282117 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-model-cache\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.282230 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.282209 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tmp-dir\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.282370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.282337 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-home\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.282431 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.282371 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6blgq\" (UniqueName: \"kubernetes.io/projected/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-kube-api-access-6blgq\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.282501 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.282476 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-dshm\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.384183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.384151 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-home\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.384405 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.384199 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6blgq\" (UniqueName: \"kubernetes.io/projected/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-kube-api-access-6blgq\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.384405 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.384236 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-dshm\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.384527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.384467 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tls-certs\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.384589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.384529 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-model-cache\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.384589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.384574 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tmp-dir\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.384696 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.384644 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-home\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.384913 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.384892 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-model-cache\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.384970 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.384944 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tmp-dir\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.388077 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.388043 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tls-certs\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.388294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.388247 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-dshm\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.410357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.407353 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6blgq\" (UniqueName: \"kubernetes.io/projected/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-kube-api-access-6blgq\") pod \"test-model-f37ca7-kserve-5979d9f6df-fxf5l\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.549341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.548546 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:40.979548 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:55:40.979512 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f4044a4_d3d1_4a5d_9861_91611fddfb8d.slice/crio-face1f2839caffaf6d56fa38e2b7d6b421c694f3669046d9808bb0a9ca47dbb0 WatchSource:0}: Error finding container face1f2839caffaf6d56fa38e2b7d6b421c694f3669046d9808bb0a9ca47dbb0: Status 404 returned error can't find the container with id face1f2839caffaf6d56fa38e2b7d6b421c694f3669046d9808bb0a9ca47dbb0 Oct 30 18:55:40.980773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:40.980745 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l"] Oct 30 18:55:41.037991 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:41.037956 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb"] Oct 30 18:55:41.042169 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:41.042147 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f21fa4c-n8vrb"] Oct 30 18:55:41.067779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:41.067755 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434920f1-3ab4-4902-8b58-a65c114c968c" path="/var/lib/kubelet/pods/434920f1-3ab4-4902-8b58-a65c114c968c/volumes" Oct 30 18:55:41.415676 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:41.415635 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" event={"ID":"9f4044a4-d3d1-4a5d-9861-91611fddfb8d","Type":"ContainerStarted","Data":"4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0"} Oct 30 18:55:41.416094 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:41.415691 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" event={"ID":"9f4044a4-d3d1-4a5d-9861-91611fddfb8d","Type":"ContainerStarted","Data":"face1f2839caffaf6d56fa38e2b7d6b421c694f3669046d9808bb0a9ca47dbb0"} Oct 30 18:55:41.454940 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:41.454898 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" podStartSLOduration=1.454884673 podStartE2EDuration="1.454884673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:41.454054181 +0000 UTC m=+3440.503099098" watchObservedRunningTime="2026-10-30 18:55:41.454884673 +0000 UTC m=+3440.503929607" Oct 30 18:55:41.549780 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:41.549755 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:49.871430 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:55:49.871392 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:55:50.549620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:50.549585 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:50.568240 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:50.568214 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:55:51.524782 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:55:51.524755 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 18:56:06.714119 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:06.714039 4862 generic.go:356] "Generic (PLEG): container finished" podID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerID="496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e" exitCode=0 Oct 30 18:56:06.714572 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:06.714116 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" event={"ID":"0c42fe79-10c6-4985-8d9e-acdd579ebcca","Type":"ContainerDied","Data":"496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e"} Oct 30 18:56:06.714616 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:06.714601 4862 scope.go:122] "RemoveContainer" containerID="496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e" Oct 30 18:56:06.716948 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:06.716915 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:56:06.717054 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:06.716992 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:56:06.717054 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:06.717028 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:56:07.725360 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:07.725315 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" event={"ID":"0c42fe79-10c6-4985-8d9e-acdd579ebcca","Type":"ContainerStarted","Data":"082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f"} Oct 30 18:56:08.181622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:08.181596 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:56:08.181796 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:08.181754 4862 patch_prober.go:29] interesting pod/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 18:56:08.181847 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:08.181797 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 18:56:09.181821 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:09.181787 4862 patch_prober.go:29] interesting pod/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 18:56:09.182222 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:09.181853 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 18:56:10.181321 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:10.181284 4862 patch_prober.go:29] interesting pod/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 18:56:10.181496 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:10.181339 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 18:56:11.181545 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:11.181512 4862 patch_prober.go:29] interesting pod/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 18:56:11.181968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:11.181561 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 18:56:12.181837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:12.181798 4862 patch_prober.go:29] interesting pod/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 18:56:12.182245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:12.181875 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 18:56:13.187809 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:13.187782 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:56:13.188306 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:13.188282 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:56:13.188895 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:13.188879 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 18:56:19.877685 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:56:19.877644 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:56:22.624778 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:22.624744 4862 scope.go:122] "RemoveContainer" containerID="99f6560f52c26b05a89757a99c599b6cefe88c83882e83a87ef04c86d40b7d88" Oct 30 18:56:36.594256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:36.594219 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:56:45.447548 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.447514 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6"] Oct 30 18:56:45.452449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.452433 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:56:45.472257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.472222 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9f6b7c\"" Oct 30 18:56:45.484868 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.484828 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9f6b7c-dockercfg-pntdx\"" Oct 30 18:56:45.489996 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.489971 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6"] Oct 30 18:56:45.647285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.642587 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6\" (UID: \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:56:45.647285 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.642654 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lpc\" (UniqueName: \"kubernetes.io/projected/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-kube-api-access-89lpc\") pod \"payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6\" (UID: \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:56:45.756818 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.756724 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6\" (UID: \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:56:45.757824 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.757783 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lpc\" (UniqueName: \"kubernetes.io/projected/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-kube-api-access-89lpc\") pod \"payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6\" (UID: \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:56:45.759936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.759914 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6\" (UID: \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:56:45.771460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.771282 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt"] Oct 30 18:56:45.777975 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.777955 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:45.792593 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.792550 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt"] Oct 30 18:56:45.794106 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.794081 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lpc\" (UniqueName: \"kubernetes.io/projected/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-kube-api-access-89lpc\") pod \"payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6\" (UID: \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:56:45.859917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.859889 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tncn\" (UniqueName: \"kubernetes.io/projected/3425a122-a8c3-4339-9665-a4df3b7da91c-kube-api-access-9tncn\") pod \"payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt\" (UID: \"3425a122-a8c3-4339-9665-a4df3b7da91c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:45.860078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.859965 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3425a122-a8c3-4339-9665-a4df3b7da91c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt\" (UID: \"3425a122-a8c3-4339-9665-a4df3b7da91c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:45.961493 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.961462 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tncn\" (UniqueName: \"kubernetes.io/projected/3425a122-a8c3-4339-9665-a4df3b7da91c-kube-api-access-9tncn\") pod \"payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt\" (UID: \"3425a122-a8c3-4339-9665-a4df3b7da91c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:45.961687 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.961511 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3425a122-a8c3-4339-9665-a4df3b7da91c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt\" (UID: \"3425a122-a8c3-4339-9665-a4df3b7da91c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:45.962203 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.962181 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3425a122-a8c3-4339-9665-a4df3b7da91c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt\" (UID: \"3425a122-a8c3-4339-9665-a4df3b7da91c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:45.982125 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:45.982092 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tncn\" (UniqueName: \"kubernetes.io/projected/3425a122-a8c3-4339-9665-a4df3b7da91c-kube-api-access-9tncn\") pod \"payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt\" (UID: \"3425a122-a8c3-4339-9665-a4df3b7da91c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:46.066562 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:46.066535 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:56:46.115928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:46.115898 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:46.384613 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:56:46.384584 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff6ec8bb_742e_46e1_80b1_01982cda2ea3.slice/crio-8ae0653a978fba0c4a4d5bb9fbb734464fbbbc789702d68f4fba7e84970ccfe6 WatchSource:0}: Error finding container 8ae0653a978fba0c4a4d5bb9fbb734464fbbbc789702d68f4fba7e84970ccfe6: Status 404 returned error can't find the container with id 8ae0653a978fba0c4a4d5bb9fbb734464fbbbc789702d68f4fba7e84970ccfe6 Oct 30 18:56:46.386226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:46.386203 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6"] Oct 30 18:56:46.471313 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:56:46.471279 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3425a122_a8c3_4339_9665_a4df3b7da91c.slice/crio-e31ced28b8b96600275c3eb11ae97eb480ac2152230607de087efcb65dc9155c WatchSource:0}: Error finding container e31ced28b8b96600275c3eb11ae97eb480ac2152230607de087efcb65dc9155c: Status 404 returned error can't find the container with id e31ced28b8b96600275c3eb11ae97eb480ac2152230607de087efcb65dc9155c Oct 30 18:56:46.472423 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:46.472385 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt"] Oct 30 18:56:47.107468 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:47.107435 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" event={"ID":"3425a122-a8c3-4339-9665-a4df3b7da91c","Type":"ContainerStarted","Data":"d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072"} Oct 30 18:56:47.107644 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:47.107494 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:47.107644 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:47.107516 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" event={"ID":"3425a122-a8c3-4339-9665-a4df3b7da91c","Type":"ContainerStarted","Data":"e31ced28b8b96600275c3eb11ae97eb480ac2152230607de087efcb65dc9155c"} Oct 30 18:56:47.109109 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:47.109079 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" event={"ID":"ff6ec8bb-742e-46e1-80b1-01982cda2ea3","Type":"ContainerStarted","Data":"751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8"} Oct 30 18:56:47.109206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:47.109126 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" event={"ID":"ff6ec8bb-742e-46e1-80b1-01982cda2ea3","Type":"ContainerStarted","Data":"8ae0653a978fba0c4a4d5bb9fbb734464fbbbc789702d68f4fba7e84970ccfe6"} Oct 30 18:56:47.109206 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:47.109155 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:56:47.159318 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:47.159257 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" podStartSLOduration=2.159245786 podStartE2EDuration="2.159245786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:47.156805401 +0000 UTC m=+3506.205850314" watchObservedRunningTime="2026-10-30 18:56:47.159245786 +0000 UTC m=+3506.208290694" Oct 30 18:56:47.202483 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:47.202439 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" podStartSLOduration=2.202426164 podStartE2EDuration="2.202426164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:47.199152037 +0000 UTC m=+3506.248196952" watchObservedRunningTime="2026-10-30 18:56:47.202426164 +0000 UTC m=+3506.251471105" Oct 30 18:56:49.871572 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:56:49.871537 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:56:57.254588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:57.254555 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7"] Oct 30 18:56:57.255101 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:57.254763 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" containerID="cri-o://006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16" gracePeriod=30 Oct 30 18:56:57.267420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:57.267390 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz"] Oct 30 18:56:57.267660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:57.267621 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" containerID="cri-o://300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2" gracePeriod=30 Oct 30 18:56:57.273086 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:57.273058 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc"] Oct 30 18:56:57.273363 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:57.273320 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" containerID="cri-o://5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8" gracePeriod=30 Oct 30 18:56:58.120089 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:58.120058 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 18:56:58.120356 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:56:58.120340 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 18:57:02.621299 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.620544 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:57:02.726358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.726325 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:57:02.732323 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.731010 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:57:02.742714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.742683 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-data\") pod \"51436817-f5b6-433f-afd3-a472760940a6\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " Oct 30 18:57:02.742898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.742736 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-envoy\") pod \"51436817-f5b6-433f-afd3-a472760940a6\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " Oct 30 18:57:02.742898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.742801 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-socket\") pod \"51436817-f5b6-433f-afd3-a472760940a6\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " Oct 30 18:57:02.743035 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.742943 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-istio-token\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-istio-token\") pod \"51436817-f5b6-433f-afd3-a472760940a6\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " Oct 30 18:57:02.743035 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.742972 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51436817-f5b6-433f-afd3-a472760940a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51436817-f5b6-433f-afd3-a472760940a6-istio-podinfo\") pod \"51436817-f5b6-433f-afd3-a472760940a6\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " Oct 30 18:57:02.743160 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.743037 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-data" pod "51436817-f5b6-433f-afd3-a472760940a6" (UID: "51436817-f5b6-433f-afd3-a472760940a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.743160 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.743043 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-kube-api-access-ljssm\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-kube-api-access-ljssm\") pod \"51436817-f5b6-433f-afd3-a472760940a6\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " Oct 30 18:57:02.743305 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.743222 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51436817-f5b6-433f-afd3-a472760940a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51436817-f5b6-433f-afd3-a472760940a6-istiod-ca-cert\") pod \"51436817-f5b6-433f-afd3-a472760940a6\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " Oct 30 18:57:02.743500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.743484 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-certs\") pod \"51436817-f5b6-433f-afd3-a472760940a6\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " Oct 30 18:57:02.743588 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.743519 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-credential-socket\") pod \"51436817-f5b6-433f-afd3-a472760940a6\" (UID: \"51436817-f5b6-433f-afd3-a472760940a6\") " Oct 30 18:57:02.743671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.743648 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-socket" pod "51436817-f5b6-433f-afd3-a472760940a6" (UID: "51436817-f5b6-433f-afd3-a472760940a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.744306 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.744251 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51436817-f5b6-433f-afd3-a472760940a6-istiod-ca-cert" pod "51436817-f5b6-433f-afd3-a472760940a6" (UID: "51436817-f5b6-433f-afd3-a472760940a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:02.744430 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.744367 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-certs" pod "51436817-f5b6-433f-afd3-a472760940a6" (UID: "51436817-f5b6-433f-afd3-a472760940a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.744500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.744449 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51436817-f5b6-433f-afd3-a472760940a6-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.744500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.744471 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.744500 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.744487 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.744660 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.744565 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-credential-socket" pod "51436817-f5b6-433f-afd3-a472760940a6" (UID: "51436817-f5b6-433f-afd3-a472760940a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.747081 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.747055 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-envoy" pod "51436817-f5b6-433f-afd3-a472760940a6" (UID: "51436817-f5b6-433f-afd3-a472760940a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.747175 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.747106 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-istio-token" pod "51436817-f5b6-433f-afd3-a472760940a6" (UID: "51436817-f5b6-433f-afd3-a472760940a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:02.747831 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.747809 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-kube-api-access-ljssm" pod "51436817-f5b6-433f-afd3-a472760940a6" (UID: "51436817-f5b6-433f-afd3-a472760940a6"). InnerVolumeSpecName "kube-api-access-ljssm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:02.752240 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.752216 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51436817-f5b6-433f-afd3-a472760940a6-istio-podinfo" pod "51436817-f5b6-433f-afd3-a472760940a6" (UID: "51436817-f5b6-433f-afd3-a472760940a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845432 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istiod-ca-cert\") pod \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845482 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-token\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-token\") pod \"d88a089e-f502-464b-9a53-791d50e8c2f2\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845547 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-certs\") pod \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845582 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-data\") pod \"d88a089e-f502-464b-9a53-791d50e8c2f2\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845630 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-kube-api-access-lxtsb\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-kube-api-access-lxtsb\") pod \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845725 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-credential-socket\") pod \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845754 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-podinfo\") pod \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845785 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-credential-socket\") pod \"d88a089e-f502-464b-9a53-791d50e8c2f2\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845820 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d88a089e-f502-464b-9a53-791d50e8c2f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88a089e-f502-464b-9a53-791d50e8c2f2-istiod-ca-cert\") pod \"d88a089e-f502-464b-9a53-791d50e8c2f2\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845865 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-envoy\") pod \"d88a089e-f502-464b-9a53-791d50e8c2f2\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845901 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-podinfo\") pod \"d88a089e-f502-464b-9a53-791d50e8c2f2\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845944 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-envoy\") pod \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.845981 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-token\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-token\") pod \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846020 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-kube-api-access-9wrzn\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-kube-api-access-9wrzn\") pod \"d88a089e-f502-464b-9a53-791d50e8c2f2\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846051 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-certs\") pod \"d88a089e-f502-464b-9a53-791d50e8c2f2\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846076 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-socket\") pod \"d88a089e-f502-464b-9a53-791d50e8c2f2\" (UID: \"d88a089e-f502-464b-9a53-791d50e8c2f2\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846135 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-socket\") pod \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846201 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-data\") pod \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\" (UID: \"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea\") " Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846767 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846782 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51436817-f5b6-433f-afd3-a472760940a6-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846791 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljssm\" (UniqueName: \"kubernetes.io/projected/51436817-f5b6-433f-afd3-a472760940a6-kube-api-access-ljssm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846808 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846820 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.846828 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51436817-f5b6-433f-afd3-a472760940a6-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.847061 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-data" pod "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" (UID: "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.847366 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istiod-ca-cert" pod "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" (UID: "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.848182 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-credential-socket" pod "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" (UID: "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.848221 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-credential-socket" pod "d88a089e-f502-464b-9a53-791d50e8c2f2" (UID: "d88a089e-f502-464b-9a53-791d50e8c2f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.848382 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-certs" pod "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" (UID: "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.848561 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-data" pod "d88a089e-f502-464b-9a53-791d50e8c2f2" (UID: "d88a089e-f502-464b-9a53-791d50e8c2f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.849037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.848838 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-socket" pod "d88a089e-f502-464b-9a53-791d50e8c2f2" (UID: "d88a089e-f502-464b-9a53-791d50e8c2f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.850898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.849221 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-certs" pod "d88a089e-f502-464b-9a53-791d50e8c2f2" (UID: "d88a089e-f502-464b-9a53-791d50e8c2f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.850898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.849249 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88a089e-f502-464b-9a53-791d50e8c2f2-istiod-ca-cert" pod "d88a089e-f502-464b-9a53-791d50e8c2f2" (UID: "d88a089e-f502-464b-9a53-791d50e8c2f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:02.850898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.849570 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-socket" pod "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" (UID: "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.854636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.854587 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-podinfo" pod "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" (UID: "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:57:02.861433 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.861393 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-kube-api-access-lxtsb" pod "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" (UID: "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea"). InnerVolumeSpecName "kube-api-access-lxtsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:02.861564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.861517 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-token" pod "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" (UID: "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:02.861769 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.861743 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-podinfo" pod "d88a089e-f502-464b-9a53-791d50e8c2f2" (UID: "d88a089e-f502-464b-9a53-791d50e8c2f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:57:02.862000 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.861970 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-kube-api-access-9wrzn" pod "d88a089e-f502-464b-9a53-791d50e8c2f2" (UID: "d88a089e-f502-464b-9a53-791d50e8c2f2"). InnerVolumeSpecName "kube-api-access-9wrzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:02.862093 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.861978 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-envoy" pod "d88a089e-f502-464b-9a53-791d50e8c2f2" (UID: "d88a089e-f502-464b-9a53-791d50e8c2f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.862201 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.862180 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-token" pod "d88a089e-f502-464b-9a53-791d50e8c2f2" (UID: "d88a089e-f502-464b-9a53-791d50e8c2f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:02.862362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.862343 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-envoy" pod "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" (UID: "39f697e0-e7d3-4347-b6bf-aa4e2541f7ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:02.948830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948793 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.948830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948822 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.948830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948831 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxtsb\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-kube-api-access-lxtsb\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.948830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948840 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.948830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948849 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.948830 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948859 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948868 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88a089e-f502-464b-9a53-791d50e8c2f2-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948876 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948883 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948891 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948898 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948908 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wrzn\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-kube-api-access-9wrzn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948915 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948924 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88a089e-f502-464b-9a53-791d50e8c2f2-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948933 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948940 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948948 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:02.949241 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:02.948955 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88a089e-f502-464b-9a53-791d50e8c2f2-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:03.240664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.240573 4862 generic.go:356] "Generic (PLEG): container finished" podID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerID="006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16" exitCode=0 Oct 30 18:57:03.240664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.240629 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" Oct 30 18:57:03.240912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.240658 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" event={"ID":"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea","Type":"ContainerDied","Data":"006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16"} Oct 30 18:57:03.240912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.240712 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7" event={"ID":"39f697e0-e7d3-4347-b6bf-aa4e2541f7ea","Type":"ContainerDied","Data":"ac03dcdd6ead22074562e9aa2d966f283c950c81db332452c408c64c1230fe40"} Oct 30 18:57:03.240912 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.240747 4862 scope.go:122] "RemoveContainer" containerID="006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16" Oct 30 18:57:03.242435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.242298 4862 generic.go:356] "Generic (PLEG): container finished" podID="51436817-f5b6-433f-afd3-a472760940a6" containerID="300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2" exitCode=0 Oct 30 18:57:03.242435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.242381 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" Oct 30 18:57:03.242435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.242406 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" event={"ID":"51436817-f5b6-433f-afd3-a472760940a6","Type":"ContainerDied","Data":"300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2"} Oct 30 18:57:03.242625 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.242445 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz" event={"ID":"51436817-f5b6-433f-afd3-a472760940a6","Type":"ContainerDied","Data":"89040f8e18d45509e8068f300b183611d1ea9bd4ad9b6d866aff53bcdd234558"} Oct 30 18:57:03.244589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.244563 4862 generic.go:356] "Generic (PLEG): container finished" podID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerID="5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8" exitCode=0 Oct 30 18:57:03.244721 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.244601 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" event={"ID":"d88a089e-f502-464b-9a53-791d50e8c2f2","Type":"ContainerDied","Data":"5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8"} Oct 30 18:57:03.244721 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.244631 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" Oct 30 18:57:03.244721 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.244644 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc" event={"ID":"d88a089e-f502-464b-9a53-791d50e8c2f2","Type":"ContainerDied","Data":"1530329afbe3f05278f3421ed33cfcfebf39e47751de5b0704c99024f86095d1"} Oct 30 18:57:03.284714 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.284526 4862 scope.go:122] "RemoveContainer" containerID="006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16" Oct 30 18:57:03.315149 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:57:03.313086 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16\": container with ID starting with 006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16 not found: ID does not exist" containerID="006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16" Oct 30 18:57:03.315149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.313147 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16"} err="failed to get container status \"006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16\": rpc error: code = NotFound desc = could not find container \"006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16\": container with ID starting with 006ec83c830e30f9e663eea713fcd647327149e8147850398e1c7a8bd4c43d16 not found: ID does not exist" Oct 30 18:57:03.315149 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.313178 4862 scope.go:122] "RemoveContainer" containerID="300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2" Oct 30 18:57:03.342661 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.342621 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc"] Oct 30 18:57:03.347801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.347764 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-t7sdc"] Oct 30 18:57:03.355741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.355706 4862 scope.go:122] "RemoveContainer" containerID="300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2" Oct 30 18:57:03.358518 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:57:03.358494 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2\": container with ID starting with 300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2 not found: ID does not exist" containerID="300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2" Oct 30 18:57:03.358637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.358528 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2"} err="failed to get container status \"300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2\": rpc error: code = NotFound desc = could not find container \"300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2\": container with ID starting with 300ea41af044bca94ede2a10591424417f35cce2db05cac3c836594fd93debb2 not found: ID does not exist" Oct 30 18:57:03.358637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.358565 4862 scope.go:122] "RemoveContainer" containerID="5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8" Oct 30 18:57:03.371825 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.371802 4862 scope.go:122] "RemoveContainer" containerID="5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8" Oct 30 18:57:03.372101 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:57:03.372080 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8\": container with ID starting with 5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8 not found: ID does not exist" containerID="5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8" Oct 30 18:57:03.372171 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.372109 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8"} err="failed to get container status \"5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8\": rpc error: code = NotFound desc = could not find container \"5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8\": container with ID starting with 5165fac270bf13d1eb873c2dc58b0d9a342373a069be6d5a0cd6380bc388c1d8 not found: ID does not exist" Oct 30 18:57:03.384180 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.384155 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz"] Oct 30 18:57:03.388531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.388509 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7jqjz"] Oct 30 18:57:03.422872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.422830 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7"] Oct 30 18:57:03.427250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.427230 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-wwft7"] Oct 30 18:57:03.750615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.750580 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml"] Oct 30 18:57:03.751564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.751542 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" Oct 30 18:57:03.751564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.751564 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" Oct 30 18:57:03.751759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.751616 4862 cpu_manager.go:414] "containerMap: removing container" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" Oct 30 18:57:03.751759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.751624 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" Oct 30 18:57:03.751759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.751652 4862 cpu_manager.go:414] "containerMap: removing container" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" Oct 30 18:57:03.751759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.751659 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" Oct 30 18:57:03.751973 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.751851 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" containerName="istio-proxy" Oct 30 18:57:03.751973 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.751874 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="51436817-f5b6-433f-afd3-a472760940a6" containerName="istio-proxy" Oct 30 18:57:03.751973 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.751888 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" containerName="istio-proxy" Oct 30 18:57:03.757527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.757504 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.774312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.772845 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f37ca7-openshift-default-dockercfg-fgcsx\"" Oct 30 18:57:03.778670 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.778648 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml"] Oct 30 18:57:03.897361 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.897331 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-envoy\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.897361 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.897361 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istiod-ca-cert\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.897556 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.897459 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-podinfo\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.897628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.897607 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-credential-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.897728 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.897712 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-data\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.897767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.897752 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrjx\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-kube-api-access-fmrjx\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.897804 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.897797 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.897837 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.897831 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-certs\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.897932 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.897919 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-token\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999120 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999089 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-podinfo\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999176 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-credential-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999224 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-data\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999293 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999254 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrjx\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-kube-api-access-fmrjx\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999319 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999347 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-certs\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999403 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-token\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999624 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999474 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-envoy\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999624 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999496 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istiod-ca-cert\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999749 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999661 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-data\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999749 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999690 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-credential-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999860 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999755 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-socket\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:03.999998 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:03.999978 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-certs\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:04.000334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.000318 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istiod-ca-cert\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:04.001811 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.001762 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-podinfo\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:04.001918 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.001865 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-envoy\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:04.018617 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.018598 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-token\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:04.018921 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.018895 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrjx\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-kube-api-access-fmrjx\") pod \"e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:04.083828 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.082589 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:04.522190 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:57:04.522154 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e18bf6_97c4_46a5_83a6_1f7ec21655f1.slice/crio-3e55913dda9d47a7627a41389d4d9a56a1f7f453cf7ee436f3b9e79b745ef08c WatchSource:0}: Error finding container 3e55913dda9d47a7627a41389d4d9a56a1f7f453cf7ee436f3b9e79b745ef08c: Status 404 returned error can't find the container with id 3e55913dda9d47a7627a41389d4d9a56a1f7f453cf7ee436f3b9e79b745ef08c Oct 30 18:57:04.524134 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.524104 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml"] Oct 30 18:57:04.525965 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.525927 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:57:04.526089 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.525992 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:57:04.526089 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:04.526022 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:57:05.065072 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:05.065037 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f697e0-e7d3-4347-b6bf-aa4e2541f7ea" path="/var/lib/kubelet/pods/39f697e0-e7d3-4347-b6bf-aa4e2541f7ea/volumes" Oct 30 18:57:05.065638 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:05.065622 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51436817-f5b6-433f-afd3-a472760940a6" path="/var/lib/kubelet/pods/51436817-f5b6-433f-afd3-a472760940a6/volumes" Oct 30 18:57:05.066082 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:05.066069 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88a089e-f502-464b-9a53-791d50e8c2f2" path="/var/lib/kubelet/pods/d88a089e-f502-464b-9a53-791d50e8c2f2/volumes" Oct 30 18:57:05.274916 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:05.274882 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" event={"ID":"90e18bf6-97c4-46a5-83a6-1f7ec21655f1","Type":"ContainerStarted","Data":"a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6"} Oct 30 18:57:05.275066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:05.274938 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" event={"ID":"90e18bf6-97c4-46a5-83a6-1f7ec21655f1","Type":"ContainerStarted","Data":"3e55913dda9d47a7627a41389d4d9a56a1f7f453cf7ee436f3b9e79b745ef08c"} Oct 30 18:57:05.319335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:05.319232 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" podStartSLOduration=2.31921867 podStartE2EDuration="2.31921867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:05.317216254 +0000 UTC m=+3524.366261167" watchObservedRunningTime="2026-10-30 18:57:05.31921867 +0000 UTC m=+3524.368263586" Oct 30 18:57:06.084328 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:06.084295 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:06.086723 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:06.086698 4862 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:57:06.086834 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:06.086744 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 18:57:07.084257 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:07.084223 4862 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:57:07.084426 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:07.084299 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 18:57:08.084722 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:08.084683 4862 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:57:08.085141 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:08.084736 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 18:57:09.083848 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:09.083816 4862 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:57:09.084034 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:09.083867 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 18:57:10.083329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:10.083254 4862 patch_prober.go:29] interesting pod/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 18:57:10.083834 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:10.083361 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 18:57:11.090367 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:11.090338 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:11.090867 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:11.090856 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:11.092370 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:11.092340 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 18:57:18.351883 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.351847 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz"] Oct 30 18:57:18.358189 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.358159 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.387808 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.387782 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz"] Oct 30 18:57:18.426083 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.426043 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-daf3ba-openshift-default-dockercfg-ztv66\"" Oct 30 18:57:18.504972 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.504926 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32dd5fc-27df-453f-b026-3811e7fac2f1-istiod-ca-cert\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.505158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.505084 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-credential-socket\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.505158 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.505137 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-token\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.505366 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.505318 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgzgg\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-kube-api-access-lgzgg\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.505432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.505373 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-envoy\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.505499 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.505470 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-data\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.505566 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.505518 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-socket\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.505644 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.505629 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-podinfo\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.505703 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.505675 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-certs\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.606999 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.606903 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-envoy\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.606999 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.606993 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-data\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.607207 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.607045 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-socket\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.607207 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.607197 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-podinfo\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.607343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.607252 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-certs\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.607343 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.607340 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32dd5fc-27df-453f-b026-3811e7fac2f1-istiod-ca-cert\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.607476 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.607445 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-credential-socket\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.607535 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.607488 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-token\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.610827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.607599 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzgg\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-kube-api-access-lgzgg\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.610827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.607637 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-socket\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.610827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.608348 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-certs\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.610827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.608366 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32dd5fc-27df-453f-b026-3811e7fac2f1-istiod-ca-cert\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.610827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.608835 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-data\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.610827 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.608908 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-credential-socket\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.612213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.612094 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-podinfo\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.672793 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.672755 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-envoy\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.672986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.672754 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-token\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.707617 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.707585 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzgg\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-kube-api-access-lgzgg\") pod \"e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:18.988416 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:18.988337 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:19.325146 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:19.325120 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz"] Oct 30 18:57:19.330847 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:57:19.327719 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32dd5fc_27df_453f_b026_3811e7fac2f1.slice/crio-672c00ca881f6262191c6a142ffcebd3bd456a0428d1434b87e8103954c93d6a WatchSource:0}: Error finding container 672c00ca881f6262191c6a142ffcebd3bd456a0428d1434b87e8103954c93d6a: Status 404 returned error can't find the container with id 672c00ca881f6262191c6a142ffcebd3bd456a0428d1434b87e8103954c93d6a Oct 30 18:57:19.334733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:19.334696 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:57:19.334842 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:19.334770 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:57:19.334842 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:19.334801 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:57:19.449725 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:19.449690 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" event={"ID":"a32dd5fc-27df-453f-b026-3811e7fac2f1","Type":"ContainerStarted","Data":"860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1"} Oct 30 18:57:19.450197 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:19.449745 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" event={"ID":"a32dd5fc-27df-453f-b026-3811e7fac2f1","Type":"ContainerStarted","Data":"672c00ca881f6262191c6a142ffcebd3bd456a0428d1434b87e8103954c93d6a"} Oct 30 18:57:19.494664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:19.494582 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" podStartSLOduration=1.49456242 podStartE2EDuration="1.49456242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:19.491152351 +0000 UTC m=+3538.540197266" watchObservedRunningTime="2026-10-30 18:57:19.49456242 +0000 UTC m=+3538.543607332" Oct 30 18:57:19.876930 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:57:19.876892 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:57:19.988439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:19.988407 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:20.995014 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:20.994983 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:21.465575 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:21.465545 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:21.466466 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:21.466448 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:57:22.657007 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:22.656969 4862 scope.go:122] "RemoveContainer" containerID="a07a8a1725128cd0e2b2c5bee4838eabc2ecc0fd2b00dd34747f8756f5b08eca" Oct 30 18:57:32.846978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:32.846904 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:57:36.878480 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:36.878442 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26"] Oct 30 18:57:36.883864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:36.883840 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:36.901303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:36.901237 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26"] Oct 30 18:57:36.908779 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:36.908759 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:57:36.908907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:36.908824 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:57:37.013181 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.013153 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a70db8-h4b26\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.013362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.013206 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjjp\" (UniqueName: \"kubernetes.io/projected/ff36e59a-d24d-413c-80aa-d937ec1279ac-kube-api-access-rcjjp\") pod \"maas-api-revoke-sub-f8a70db8-h4b26\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.013445 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.013425 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a70db8-h4b26\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.114596 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.114533 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a70db8-h4b26\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.114849 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.114829 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a70db8-h4b26\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.114949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.114887 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjjp\" (UniqueName: \"kubernetes.io/projected/ff36e59a-d24d-413c-80aa-d937ec1279ac-kube-api-access-rcjjp\") pod \"maas-api-revoke-sub-f8a70db8-h4b26\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.115564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.115544 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a70db8-h4b26\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.117662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.117606 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a70db8-h4b26\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.135204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.135079 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjjp\" (UniqueName: \"kubernetes.io/projected/ff36e59a-d24d-413c-80aa-d937ec1279ac-kube-api-access-rcjjp\") pod \"maas-api-revoke-sub-f8a70db8-h4b26\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.200012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.199989 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:37.505859 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:57:37.505831 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff36e59a_d24d_413c_80aa_d937ec1279ac.slice/crio-df5847a004940c68e26ccd50bf660f51239776b322146c6266085bbaad911566 WatchSource:0}: Error finding container df5847a004940c68e26ccd50bf660f51239776b322146c6266085bbaad911566: Status 404 returned error can't find the container with id df5847a004940c68e26ccd50bf660f51239776b322146c6266085bbaad911566 Oct 30 18:57:37.507885 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.507859 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26"] Oct 30 18:57:37.611098 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:37.611069 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" event={"ID":"ff36e59a-d24d-413c-80aa-d937ec1279ac","Type":"ContainerStarted","Data":"df5847a004940c68e26ccd50bf660f51239776b322146c6266085bbaad911566"} Oct 30 18:57:38.621274 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:38.621223 4862 generic.go:356] "Generic (PLEG): container finished" podID="ff36e59a-d24d-413c-80aa-d937ec1279ac" containerID="b2d51e6c03d9bee13790279574acff08d4baa6101a6460c506d6efbc62179c29" exitCode=0 Oct 30 18:57:38.621693 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:38.621297 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" event={"ID":"ff36e59a-d24d-413c-80aa-d937ec1279ac","Type":"ContainerDied","Data":"b2d51e6c03d9bee13790279574acff08d4baa6101a6460c506d6efbc62179c29"} Oct 30 18:57:39.813926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:39.813896 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:39.963147 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:39.963063 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-xks-ca\") pod \"ff36e59a-d24d-413c-80aa-d937ec1279ac\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " Oct 30 18:57:39.963347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:39.963234 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-openshift-ca\") pod \"ff36e59a-d24d-413c-80aa-d937ec1279ac\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " Oct 30 18:57:39.963450 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:39.963436 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff36e59a-d24d-413c-80aa-d937ec1279ac-kube-api-access-rcjjp\" (UniqueName: \"kubernetes.io/projected/ff36e59a-d24d-413c-80aa-d937ec1279ac-kube-api-access-rcjjp\") pod \"ff36e59a-d24d-413c-80aa-d937ec1279ac\" (UID: \"ff36e59a-d24d-413c-80aa-d937ec1279ac\") " Oct 30 18:57:39.963610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:39.963582 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-openshift-ca" pod "ff36e59a-d24d-413c-80aa-d937ec1279ac" (UID: "ff36e59a-d24d-413c-80aa-d937ec1279ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:39.976282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:39.964604 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:39.976282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:39.965886 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-xks-ca" pod "ff36e59a-d24d-413c-80aa-d937ec1279ac" (UID: "ff36e59a-d24d-413c-80aa-d937ec1279ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:39.979439 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:39.979413 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff36e59a-d24d-413c-80aa-d937ec1279ac-kube-api-access-rcjjp" pod "ff36e59a-d24d-413c-80aa-d937ec1279ac" (UID: "ff36e59a-d24d-413c-80aa-d937ec1279ac"). InnerVolumeSpecName "kube-api-access-rcjjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:40.065917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:40.065892 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcjjp\" (UniqueName: \"kubernetes.io/projected/ff36e59a-d24d-413c-80aa-d937ec1279ac-kube-api-access-rcjjp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:40.065917 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:40.065917 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff36e59a-d24d-413c-80aa-d937ec1279ac-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:40.639506 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:40.639480 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" Oct 30 18:57:40.639708 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:40.639513 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26" event={"ID":"ff36e59a-d24d-413c-80aa-d937ec1279ac","Type":"ContainerDied","Data":"df5847a004940c68e26ccd50bf660f51239776b322146c6266085bbaad911566"} Oct 30 18:57:40.639708 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:40.639563 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df5847a004940c68e26ccd50bf660f51239776b322146c6266085bbaad911566" Oct 30 18:57:45.344490 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:45.344457 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg"] Oct 30 18:57:45.344881 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:45.344647 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerName="istio-proxy" containerID="cri-o://f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1" gracePeriod=30 Oct 30 18:57:47.938148 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:47.938115 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk"] Oct 30 18:57:48.691924 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.691842 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk"] Oct 30 18:57:48.692700 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.692679 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ff36e59a-d24d-413c-80aa-d937ec1279ac" containerName="revoke-keys" Oct 30 18:57:48.692700 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.692697 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff36e59a-d24d-413c-80aa-d937ec1279ac" containerName="revoke-keys" Oct 30 18:57:48.692898 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.692840 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff36e59a-d24d-413c-80aa-d937ec1279ac" containerName="revoke-keys" Oct 30 18:57:48.696619 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.696602 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:48.715822 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.715799 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk"] Oct 30 18:57:48.722527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.722498 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:57:48.722636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.722564 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:57:48.744481 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.744455 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862387e2-wr2bk\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:48.744766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.744739 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm2l6\" (UniqueName: \"kubernetes.io/projected/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-kube-api-access-lm2l6\") pod \"maas-api-revoke-sub-862387e2-wr2bk\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:48.744859 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.744810 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862387e2-wr2bk\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:48.846049 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.846013 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm2l6\" (UniqueName: \"kubernetes.io/projected/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-kube-api-access-lm2l6\") pod \"maas-api-revoke-sub-862387e2-wr2bk\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:48.846232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.846069 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862387e2-wr2bk\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:48.846232 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.846176 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862387e2-wr2bk\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:48.846832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.846807 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862387e2-wr2bk\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:48.848745 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.848720 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862387e2-wr2bk\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:48.866256 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:48.866229 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm2l6\" (UniqueName: \"kubernetes.io/projected/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-kube-api-access-lm2l6\") pod \"maas-api-revoke-sub-862387e2-wr2bk\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:49.013379 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:49.013304 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:49.326709 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:57:49.326678 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f6d1a9_c05c_4f9a_9057_8c6d94609e8d.slice/crio-56674fac408e704184edead917b89552ce1e1634ea5623b658a029dbe9377a25 WatchSource:0}: Error finding container 56674fac408e704184edead917b89552ce1e1634ea5623b658a029dbe9377a25: Status 404 returned error can't find the container with id 56674fac408e704184edead917b89552ce1e1634ea5623b658a029dbe9377a25 Oct 30 18:57:49.327216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:49.327188 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk"] Oct 30 18:57:49.714850 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:49.714777 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" event={"ID":"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d","Type":"ContainerStarted","Data":"56674fac408e704184edead917b89552ce1e1634ea5623b658a029dbe9377a25"} Oct 30 18:57:49.873500 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:57:49.873438 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:57:50.636458 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.636436 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:57:50.670969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.670943 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-token\") pod \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " Oct 30 18:57:50.671122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671011 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-socket\") pod \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " Oct 30 18:57:50.671122 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671115 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-credential-socket\") pod \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " Oct 30 18:57:50.671255 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671142 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-certs\") pod \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " Oct 30 18:57:50.671368 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671316 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-socket" pod "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" (UID: "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:50.671368 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671329 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istiod-ca-cert\") pod \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " Oct 30 18:57:50.671502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671379 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-kube-api-access-9gfgj\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-kube-api-access-9gfgj\") pod \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " Oct 30 18:57:50.671502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671418 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-credential-socket" pod "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" (UID: "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:50.671502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671438 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-certs" pod "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" (UID: "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:50.671502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671453 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-envoy\") pod \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " Oct 30 18:57:50.671502 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671486 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-data\") pod \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " Oct 30 18:57:50.671763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671513 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-podinfo\") pod \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\" (UID: \"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a\") " Oct 30 18:57:50.671763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671710 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-data" pod "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" (UID: "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:50.671763 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.671712 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istiod-ca-cert" pod "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" (UID: "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:50.672205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.672184 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:50.672205 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.672205 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:50.672571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.672214 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:50.672571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.672227 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:50.672571 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.672236 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:50.673508 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.673478 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-token" pod "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" (UID: "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:50.673926 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.673902 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-podinfo" pod "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" (UID: "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:57:50.674195 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.674173 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-envoy" pod "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" (UID: "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:50.674277 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.674210 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-kube-api-access-9gfgj" pod "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" (UID: "a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a"). InnerVolumeSpecName "kube-api-access-9gfgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:50.731764 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.731707 4862 generic.go:356] "Generic (PLEG): container finished" podID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerID="f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1" exitCode=0 Oct 30 18:57:50.731937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.731785 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" event={"ID":"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a","Type":"ContainerDied","Data":"f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1"} Oct 30 18:57:50.731937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.731856 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" event={"ID":"a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a","Type":"ContainerDied","Data":"475afdf52929900af3919e48db3c6543e23c15a0b1fba8d762ecf103710e91eb"} Oct 30 18:57:50.731937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.731884 4862 scope.go:122] "RemoveContainer" containerID="f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1" Oct 30 18:57:50.732140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.732073 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg" Oct 30 18:57:50.734531 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.734503 4862 generic.go:356] "Generic (PLEG): container finished" podID="40f6d1a9-c05c-4f9a-9057-8c6d94609e8d" containerID="b7186296ea7758dc169ca0b3c6b3ca3e4569399840fc398edfefc1f689e4a3cc" exitCode=0 Oct 30 18:57:50.734751 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.734730 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" event={"ID":"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d","Type":"ContainerDied","Data":"b7186296ea7758dc169ca0b3c6b3ca3e4569399840fc398edfefc1f689e4a3cc"} Oct 30 18:57:50.745118 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.745101 4862 scope.go:122] "RemoveContainer" containerID="f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1" Oct 30 18:57:50.745389 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:57:50.745371 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1\": container with ID starting with f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1 not found: ID does not exist" containerID="f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1" Oct 30 18:57:50.745465 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.745398 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1"} err="failed to get container status \"f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1\": rpc error: code = NotFound desc = could not find container \"f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1\": container with ID starting with f83f417be2dc80efe3a81d8484c60fecd08f5a3ff8fe8ad48a43bf29e7f61eb1 not found: ID does not exist" Oct 30 18:57:50.773290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.773247 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gfgj\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-kube-api-access-9gfgj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:50.773290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.773284 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:50.773290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.773294 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:50.773474 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.773302 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:50.792126 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.792106 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg"] Oct 30 18:57:50.801228 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:50.801207 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8862aa-openshift-default-65c476f4d7-x6mxg"] Oct 30 18:57:51.067204 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:51.067162 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" path="/var/lib/kubelet/pods/a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a/volumes" Oct 30 18:57:51.922314 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:51.922290 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:51.989514 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:51.989482 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-xks-ca\") pod \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " Oct 30 18:57:51.989690 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:51.989649 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-openshift-ca\") pod \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " Oct 30 18:57:51.989821 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:51.989802 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-kube-api-access-lm2l6\" (UniqueName: \"kubernetes.io/projected/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-kube-api-access-lm2l6\") pod \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\" (UID: \"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d\") " Oct 30 18:57:51.990250 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:51.990223 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-openshift-ca" pod "40f6d1a9-c05c-4f9a-9057-8c6d94609e8d" (UID: "40f6d1a9-c05c-4f9a-9057-8c6d94609e8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:51.990767 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:51.990743 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:51.992386 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:51.992360 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-xks-ca" pod "40f6d1a9-c05c-4f9a-9057-8c6d94609e8d" (UID: "40f6d1a9-c05c-4f9a-9057-8c6d94609e8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:51.993030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:51.993004 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-kube-api-access-lm2l6" pod "40f6d1a9-c05c-4f9a-9057-8c6d94609e8d" (UID: "40f6d1a9-c05c-4f9a-9057-8c6d94609e8d"). InnerVolumeSpecName "kube-api-access-lm2l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:52.092220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.092194 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:52.092220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.092218 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm2l6\" (UniqueName: \"kubernetes.io/projected/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d-kube-api-access-lm2l6\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:57:52.753549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.753521 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" Oct 30 18:57:52.753727 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.753552 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk" event={"ID":"40f6d1a9-c05c-4f9a-9057-8c6d94609e8d","Type":"ContainerDied","Data":"56674fac408e704184edead917b89552ce1e1634ea5623b658a029dbe9377a25"} Oct 30 18:57:52.753727 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.753626 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56674fac408e704184edead917b89552ce1e1634ea5623b658a029dbe9377a25" Oct 30 18:57:52.821373 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.821342 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg"] Oct 30 18:57:52.822070 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.822054 4862 cpu_manager.go:414] "containerMap: removing container" podUID="40f6d1a9-c05c-4f9a-9057-8c6d94609e8d" containerName="revoke-keys" Oct 30 18:57:52.822129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.822071 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f6d1a9-c05c-4f9a-9057-8c6d94609e8d" containerName="revoke-keys" Oct 30 18:57:52.822129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.822102 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerName="istio-proxy" Oct 30 18:57:52.822129 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.822107 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerName="istio-proxy" Oct 30 18:57:52.822245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.822236 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f6d1a9-c05c-4f9a-9057-8c6d94609e8d" containerName="revoke-keys" Oct 30 18:57:52.822312 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.822252 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ccd4cf-9e7c-4fce-bf15-fe04d9b7b39a" containerName="istio-proxy" Oct 30 18:57:52.826216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.826192 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:57:52.841607 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.841588 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-daf3ba-dockercfg-tbvxd\"" Oct 30 18:57:52.841997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.841982 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-daf3ba\"" Oct 30 18:57:52.849893 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:52.849863 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg"] Oct 30 18:57:53.061290 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.059099 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6bnq\" (UniqueName: \"kubernetes.io/projected/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-kube-api-access-n6bnq\") pod \"payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg\" (UID: \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:57:53.097362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.097320 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg\" (UID: \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:57:53.122637 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.122602 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6"] Oct 30 18:57:53.128011 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.127969 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6"] Oct 30 18:57:53.128192 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.128081 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:57:53.198993 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.198962 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg\" (UID: \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:57:53.199111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.199084 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bnq\" (UniqueName: \"kubernetes.io/projected/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-kube-api-access-n6bnq\") pod \"payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg\" (UID: \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:57:53.199252 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.199233 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfdm\" (UniqueName: \"kubernetes.io/projected/af37f402-330d-45e5-8de7-1e6594732b37-kube-api-access-slfdm\") pod \"payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6\" (UID: \"af37f402-330d-45e5-8de7-1e6594732b37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:57:53.199647 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.199615 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37f402-330d-45e5-8de7-1e6594732b37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6\" (UID: \"af37f402-330d-45e5-8de7-1e6594732b37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:57:53.199969 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.199948 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg\" (UID: \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:57:53.218892 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.218862 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bnq\" (UniqueName: \"kubernetes.io/projected/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-kube-api-access-n6bnq\") pod \"payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg\" (UID: \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:57:53.301813 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.301784 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfdm\" (UniqueName: \"kubernetes.io/projected/af37f402-330d-45e5-8de7-1e6594732b37-kube-api-access-slfdm\") pod \"payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6\" (UID: \"af37f402-330d-45e5-8de7-1e6594732b37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:57:53.301959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.301934 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37f402-330d-45e5-8de7-1e6594732b37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6\" (UID: \"af37f402-330d-45e5-8de7-1e6594732b37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:57:53.302791 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.302769 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37f402-330d-45e5-8de7-1e6594732b37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6\" (UID: \"af37f402-330d-45e5-8de7-1e6594732b37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:57:53.321786 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.321728 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfdm\" (UniqueName: \"kubernetes.io/projected/af37f402-330d-45e5-8de7-1e6594732b37-kube-api-access-slfdm\") pod \"payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6\" (UID: \"af37f402-330d-45e5-8de7-1e6594732b37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:57:53.440816 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.440787 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:57:53.445938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.445454 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:57:53.774190 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:57:53.774163 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3cbe16_00bd_4ae8_9e1d_5e5420a014d1.slice/crio-6574b1a075d0013d9bebfd58ea8f60a58d25019fad671b6a533a1eb2282a5423 WatchSource:0}: Error finding container 6574b1a075d0013d9bebfd58ea8f60a58d25019fad671b6a533a1eb2282a5423: Status 404 returned error can't find the container with id 6574b1a075d0013d9bebfd58ea8f60a58d25019fad671b6a533a1eb2282a5423 Oct 30 18:57:53.775231 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.775211 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg"] Oct 30 18:57:53.789645 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:53.789617 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6"] Oct 30 18:57:53.791343 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:57:53.791319 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf37f402_330d_45e5_8de7_1e6594732b37.slice/crio-efb6ed46a0e14523d28cc655aab07b3be922ce27f656066cc54cf7dc4ba12b38 WatchSource:0}: Error finding container efb6ed46a0e14523d28cc655aab07b3be922ce27f656066cc54cf7dc4ba12b38: Status 404 returned error can't find the container with id efb6ed46a0e14523d28cc655aab07b3be922ce27f656066cc54cf7dc4ba12b38 Oct 30 18:57:54.773420 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:54.773384 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" event={"ID":"af37f402-330d-45e5-8de7-1e6594732b37","Type":"ContainerStarted","Data":"83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe"} Oct 30 18:57:54.773907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:54.773446 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" event={"ID":"af37f402-330d-45e5-8de7-1e6594732b37","Type":"ContainerStarted","Data":"efb6ed46a0e14523d28cc655aab07b3be922ce27f656066cc54cf7dc4ba12b38"} Oct 30 18:57:54.773907 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:54.773479 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:57:54.775140 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:54.775108 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" event={"ID":"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1","Type":"ContainerStarted","Data":"d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86"} Oct 30 18:57:54.775295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:54.775165 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" event={"ID":"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1","Type":"ContainerStarted","Data":"6574b1a075d0013d9bebfd58ea8f60a58d25019fad671b6a533a1eb2282a5423"} Oct 30 18:57:54.775295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:54.775195 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:57:54.821295 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:54.821237 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" podStartSLOduration=2.821224571 podStartE2EDuration="2.821224571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:54.818570552 +0000 UTC m=+3573.867615465" watchObservedRunningTime="2026-10-30 18:57:54.821224571 +0000 UTC m=+3573.870269486" Oct 30 18:57:54.863012 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:57:54.862974 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" podStartSLOduration=2.862962119 podStartE2EDuration="2.862962119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:54.858958734 +0000 UTC m=+3573.908003647" watchObservedRunningTime="2026-10-30 18:57:54.862962119 +0000 UTC m=+3573.912007031" Oct 30 18:58:02.960997 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:02.960955 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" podUID="cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" containerName="main" containerID="cri-o://4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627" gracePeriod=45 Oct 30 18:58:03.073162 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.073120 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" podUID="cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.205:8000/ready\": dial tcp 10.133.0.205:8000: connect: connection refused" Oct 30 18:58:03.261001 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.260977 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:58:03.381771 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.381736 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-model-cache\") pod \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " Oct 30 18:58:03.381968 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.381928 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tls-certs\") pod \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " Oct 30 18:58:03.382039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.382003 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-model-cache" pod "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" (UID: "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:03.382094 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.382058 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-kube-api-access-zlbr7\" (UniqueName: \"kubernetes.io/projected/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-kube-api-access-zlbr7\") pod \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " Oct 30 18:58:03.382170 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.382153 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-dshm\") pod \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " Oct 30 18:58:03.382231 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.382216 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-home\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-home\") pod \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " Oct 30 18:58:03.382310 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.382295 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tmp-dir\") pod \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\" (UID: \"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9\") " Oct 30 18:58:03.382553 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.382524 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-home" pod "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" (UID: "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:03.382801 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.382565 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tmp-dir" pod "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" (UID: "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:03.383235 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.383211 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:03.383347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.383239 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:03.383347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.383254 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:03.384549 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.384522 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-dshm" pod "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" (UID: "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:03.384662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.384596 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-kube-api-access-zlbr7" pod "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" (UID: "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9"). InnerVolumeSpecName "kube-api-access-zlbr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:03.384662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.384603 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tls-certs" pod "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" (UID: "cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:03.484463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.484380 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:03.484463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.484425 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlbr7\" (UniqueName: \"kubernetes.io/projected/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-kube-api-access-zlbr7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:03.484463 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.484436 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:03.848025 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.847983 4862 generic.go:356] "Generic (PLEG): container finished" podID="cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" containerID="4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627" exitCode=0 Oct 30 18:58:03.848325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.848022 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" event={"ID":"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9","Type":"ContainerDied","Data":"4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627"} Oct 30 18:58:03.848325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.848068 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" event={"ID":"cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9","Type":"ContainerDied","Data":"750dfe655378a74429f9df29adeabae22c1c8776dcab08173ac5b3dc4c7d7b72"} Oct 30 18:58:03.848325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.848085 4862 scope.go:122] "RemoveContainer" containerID="4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627" Oct 30 18:58:03.848325 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.848035 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk" Oct 30 18:58:03.860372 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.860350 4862 scope.go:122] "RemoveContainer" containerID="4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627" Oct 30 18:58:03.860680 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:58:03.860659 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627\": container with ID starting with 4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627 not found: ID does not exist" containerID="4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627" Oct 30 18:58:03.860755 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.860693 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627"} err="failed to get container status \"4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627\": rpc error: code = NotFound desc = could not find container \"4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627\": container with ID starting with 4f014b61e8c5c11fd06d767abcb050be63e6ca5e3b5211e98c8f90c6b7fc8627 not found: ID does not exist" Oct 30 18:58:03.902124 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.902100 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk"] Oct 30 18:58:03.932080 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:03.932054 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0cc6b1/test-model-0cc6b1-kserve-759bb69b69-j6hhk"] Oct 30 18:58:05.063805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:05.063775 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" path="/var/lib/kubelet/pods/cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9/volumes" Oct 30 18:58:05.785894 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:05.785865 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 18:58:05.786078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:05.786021 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 18:58:09.922721 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:09.922688 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj"] Oct 30 18:58:10.404589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.404554 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h"] Oct 30 18:58:10.405781 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.405760 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" containerName="main" Oct 30 18:58:10.405897 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.405783 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" containerName="main" Oct 30 18:58:10.405960 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.405950 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe93ef3-4e4b-45e5-b3ea-12ae43e224c9" containerName="main" Oct 30 18:58:10.411462 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.411438 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.427610 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.427582 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:58:10.428805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.428776 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h"] Oct 30 18:58:10.434415 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.434390 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:58:10.495773 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.495745 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bw6v\" (UniqueName: \"kubernetes.io/projected/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-kube-api-access-5bw6v\") pod \"maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.495933 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.495895 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.495986 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.495947 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.597805 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.597778 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bw6v\" (UniqueName: \"kubernetes.io/projected/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-kube-api-access-5bw6v\") pod \"maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.597978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.597886 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.597978 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.597911 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.598576 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.598551 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.600340 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.600319 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.618130 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.618103 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bw6v\" (UniqueName: \"kubernetes.io/projected/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-kube-api-access-5bw6v\") pod \"maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:10.728680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:10.728615 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:11.032037 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:11.031957 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb"] Oct 30 18:58:11.036132 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:58:11.036102 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf115e9_fad0_493f_9d2b_41e28fdd2ee6.slice/crio-fd39d49429f5c582b3d20786d538f80864fa5484cef6bad89377ad685fad3d42 WatchSource:0}: Error finding container fd39d49429f5c582b3d20786d538f80864fa5484cef6bad89377ad685fad3d42: Status 404 returned error can't find the container with id fd39d49429f5c582b3d20786d538f80864fa5484cef6bad89377ad685fad3d42 Oct 30 18:58:11.036756 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:11.036730 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h"] Oct 30 18:58:11.043459 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:11.043431 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b0b388-qkvvb"] Oct 30 18:58:11.069585 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:11.069559 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d34f511-440b-4a33-8b00-e22bf0e745db" path="/var/lib/kubelet/pods/9d34f511-440b-4a33-8b00-e22bf0e745db/volumes" Oct 30 18:58:11.954844 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:11.954811 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" event={"ID":"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6","Type":"ContainerStarted","Data":"8dcd1c44875aca8dcb2664248c47e79da149c8da011c0acd09e2121f3c95e10e"} Oct 30 18:58:11.955021 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:11.954868 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" event={"ID":"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6","Type":"ContainerStarted","Data":"fd39d49429f5c582b3d20786d538f80864fa5484cef6bad89377ad685fad3d42"} Oct 30 18:58:11.999869 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:11.999819 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" podStartSLOduration=1.999805826 podStartE2EDuration="1.999805826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:11.995677915 +0000 UTC m=+3591.044722848" watchObservedRunningTime="2026-10-30 18:58:11.999805826 +0000 UTC m=+3591.048850738" Oct 30 18:58:12.964906 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:12.964870 4862 generic.go:356] "Generic (PLEG): container finished" podID="ecf115e9-fad0-493f-9d2b-41e28fdd2ee6" containerID="8dcd1c44875aca8dcb2664248c47e79da149c8da011c0acd09e2121f3c95e10e" exitCode=0 Oct 30 18:58:12.965390 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:12.964962 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" event={"ID":"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6","Type":"ContainerDied","Data":"8dcd1c44875aca8dcb2664248c47e79da149c8da011c0acd09e2121f3c95e10e"} Oct 30 18:58:14.136358 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.136337 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:14.254432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.254362 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-kube-api-access-5bw6v\" (UniqueName: \"kubernetes.io/projected/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-kube-api-access-5bw6v\") pod \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " Oct 30 18:58:14.254432 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.254410 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-xks-ca\") pod \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " Oct 30 18:58:14.254615 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.254568 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-openshift-ca\") pod \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\" (UID: \"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6\") " Oct 30 18:58:14.254970 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.254936 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-openshift-ca" pod "ecf115e9-fad0-493f-9d2b-41e28fdd2ee6" (UID: "ecf115e9-fad0-493f-9d2b-41e28fdd2ee6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:14.255421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.255403 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:14.256643 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.256623 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-xks-ca" pod "ecf115e9-fad0-493f-9d2b-41e28fdd2ee6" (UID: "ecf115e9-fad0-493f-9d2b-41e28fdd2ee6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:14.256759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.256649 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-kube-api-access-5bw6v" pod "ecf115e9-fad0-493f-9d2b-41e28fdd2ee6" (UID: "ecf115e9-fad0-493f-9d2b-41e28fdd2ee6"). InnerVolumeSpecName "kube-api-access-5bw6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:14.357114 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.357083 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bw6v\" (UniqueName: \"kubernetes.io/projected/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-kube-api-access-5bw6v\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:14.357114 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:14.357115 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:15.005239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:15.005197 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" event={"ID":"ecf115e9-fad0-493f-9d2b-41e28fdd2ee6","Type":"ContainerDied","Data":"fd39d49429f5c582b3d20786d538f80864fa5484cef6bad89377ad685fad3d42"} Oct 30 18:58:15.005487 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:15.005327 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd39d49429f5c582b3d20786d538f80864fa5484cef6bad89377ad685fad3d42" Oct 30 18:58:15.005487 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:15.005413 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h" Oct 30 18:58:19.872730 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:58:19.872694 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:58:20.897632 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:20.897591 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2"] Oct 30 18:58:20.929245 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:20.929213 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j"] Oct 30 18:58:21.039454 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:21.039408 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq"] Oct 30 18:58:21.046345 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:21.046315 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-489d6b99-849gq"] Oct 30 18:58:21.068671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:21.068319 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d048b581-ed21-4e8d-bff2-d45ff2a72e1f" path="/var/lib/kubelet/pods/d048b581-ed21-4e8d-bff2-d45ff2a72e1f/volumes" Oct 30 18:58:21.872527 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:21.872228 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:58:21.885430 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:21.885403 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 18:58:24.961456 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:24.961413 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" podUID="b5fafa9f-cc81-422f-a01f-b381778d26a3" containerName="main" containerID="cri-o://f7459ebbb195d2f7f713bd31a61c5038c27c377eb04db88a203f9f5424faa902" gracePeriod=45 Oct 30 18:58:25.091724 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.091689 4862 generic.go:356] "Generic (PLEG): container finished" podID="b5fafa9f-cc81-422f-a01f-b381778d26a3" containerID="f7459ebbb195d2f7f713bd31a61c5038c27c377eb04db88a203f9f5424faa902" exitCode=0 Oct 30 18:58:25.091864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.091755 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" event={"ID":"b5fafa9f-cc81-422f-a01f-b381778d26a3","Type":"ContainerDied","Data":"f7459ebbb195d2f7f713bd31a61c5038c27c377eb04db88a203f9f5424faa902"} Oct 30 18:58:25.273031 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.273000 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:58:25.425214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.425183 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-tmp-dir\") pod \"b5fafa9f-cc81-422f-a01f-b381778d26a3\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " Oct 30 18:58:25.425419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.425223 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-dshm\") pod \"b5fafa9f-cc81-422f-a01f-b381778d26a3\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " Oct 30 18:58:25.425419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.425313 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5fafa9f-cc81-422f-a01f-b381778d26a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5fafa9f-cc81-422f-a01f-b381778d26a3-tls-certs\") pod \"b5fafa9f-cc81-422f-a01f-b381778d26a3\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " Oct 30 18:58:25.425419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.425355 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fafa9f-cc81-422f-a01f-b381778d26a3-kube-api-access-76jh7\" (UniqueName: \"kubernetes.io/projected/b5fafa9f-cc81-422f-a01f-b381778d26a3-kube-api-access-76jh7\") pod \"b5fafa9f-cc81-422f-a01f-b381778d26a3\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " Oct 30 18:58:25.425603 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.425516 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-tmp-dir" pod "b5fafa9f-cc81-422f-a01f-b381778d26a3" (UID: "b5fafa9f-cc81-422f-a01f-b381778d26a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:25.425654 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.425643 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-model-cache\") pod \"b5fafa9f-cc81-422f-a01f-b381778d26a3\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " Oct 30 18:58:25.425746 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.425732 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-home\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-home\") pod \"b5fafa9f-cc81-422f-a01f-b381778d26a3\" (UID: \"b5fafa9f-cc81-422f-a01f-b381778d26a3\") " Oct 30 18:58:25.426039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.425917 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-model-cache" pod "b5fafa9f-cc81-422f-a01f-b381778d26a3" (UID: "b5fafa9f-cc81-422f-a01f-b381778d26a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:25.426039 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.425989 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-home" pod "b5fafa9f-cc81-422f-a01f-b381778d26a3" (UID: "b5fafa9f-cc81-422f-a01f-b381778d26a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:25.426651 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.426629 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.426741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.426657 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.426741 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.426674 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.427832 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.427807 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5fafa9f-cc81-422f-a01f-b381778d26a3-tls-certs" pod "b5fafa9f-cc81-422f-a01f-b381778d26a3" (UID: "b5fafa9f-cc81-422f-a01f-b381778d26a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:25.427931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.427873 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fafa9f-cc81-422f-a01f-b381778d26a3-kube-api-access-76jh7" pod "b5fafa9f-cc81-422f-a01f-b381778d26a3" (UID: "b5fafa9f-cc81-422f-a01f-b381778d26a3"). InnerVolumeSpecName "kube-api-access-76jh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:25.427931 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.427879 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-dshm" pod "b5fafa9f-cc81-422f-a01f-b381778d26a3" (UID: "b5fafa9f-cc81-422f-a01f-b381778d26a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:25.528533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.528474 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5fafa9f-cc81-422f-a01f-b381778d26a3-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.528533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.528497 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5fafa9f-cc81-422f-a01f-b381778d26a3-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.528533 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.528506 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76jh7\" (UniqueName: \"kubernetes.io/projected/b5fafa9f-cc81-422f-a01f-b381778d26a3-kube-api-access-76jh7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.898413 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.898369 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" podUID="5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" containerName="payload-processing" containerID="cri-o://8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c" gracePeriod=25 Oct 30 18:58:25.929612 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:25.929574 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" podUID="3e9466f2-fd67-4562-b757-8a43a6757a03" containerName="payload-pre-processing" containerID="cri-o://00a4abacc5e63b8b5051c41b8969cd95fc278a5f82d4ad52e897913c4a656768" gracePeriod=25 Oct 30 18:58:26.101309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:26.101275 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" Oct 30 18:58:26.101309 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:26.101291 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj" event={"ID":"b5fafa9f-cc81-422f-a01f-b381778d26a3","Type":"ContainerDied","Data":"bf8b72561dbe5b51cff557702b1afdb3dddcdde93a2cef29d562f20ea164aa0a"} Oct 30 18:58:26.101790 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:26.101345 4862 scope.go:122] "RemoveContainer" containerID="f7459ebbb195d2f7f713bd31a61c5038c27c377eb04db88a203f9f5424faa902" Oct 30 18:58:26.155864 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:26.155791 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj"] Oct 30 18:58:26.167604 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:26.167576 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c52ec7/test-model-c52ec7-kserve-766594dcfc-rm5vj"] Oct 30 18:58:27.066405 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:27.066371 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5fafa9f-cc81-422f-a01f-b381778d26a3" path="/var/lib/kubelet/pods/b5fafa9f-cc81-422f-a01f-b381778d26a3/volumes" Oct 30 18:58:31.735749 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:31.735705 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" podUID="5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:58:31.736127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:31.735764 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" podUID="3e9466f2-fd67-4562-b757-8a43a6757a03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:58:34.045954 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:34.045909 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb"] Oct 30 18:58:34.053473 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:34.053435 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7786d5e7-wszfb"] Oct 30 18:58:35.063983 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:35.063951 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c9e4a0-c531-4985-ab09-54895977c7dc" path="/var/lib/kubelet/pods/c0c9e4a0-c531-4985-ab09-54895977c7dc/volumes" Oct 30 18:58:41.735704 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:41.735669 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" podUID="5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:58:41.735704 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:41.735692 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" podUID="3e9466f2-fd67-4562-b757-8a43a6757a03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:58:46.246294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.246251 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:58:46.262452 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.262426 4862 generic.go:356] "Generic (PLEG): container finished" podID="3e9466f2-fd67-4562-b757-8a43a6757a03" containerID="00a4abacc5e63b8b5051c41b8969cd95fc278a5f82d4ad52e897913c4a656768" exitCode=0 Oct 30 18:58:46.262600 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.262500 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" event={"ID":"3e9466f2-fd67-4562-b757-8a43a6757a03","Type":"ContainerDied","Data":"00a4abacc5e63b8b5051c41b8969cd95fc278a5f82d4ad52e897913c4a656768"} Oct 30 18:58:46.264052 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.264030 4862 generic.go:356] "Generic (PLEG): container finished" podID="5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" containerID="8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c" exitCode=0 Oct 30 18:58:46.264183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.264074 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" event={"ID":"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9","Type":"ContainerDied","Data":"8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c"} Oct 30 18:58:46.264183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.264086 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" Oct 30 18:58:46.264183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.264108 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2" event={"ID":"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9","Type":"ContainerDied","Data":"f9a0eb6d920128a26953d8634eba87751369305973ee15a00d57ffcbf044466a"} Oct 30 18:58:46.264183 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.264129 4862 scope.go:122] "RemoveContainer" containerID="8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c" Oct 30 18:58:46.303449 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.302893 4862 scope.go:122] "RemoveContainer" containerID="8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c" Oct 30 18:58:46.303846 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:58:46.303817 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c\": container with ID starting with 8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c not found: ID does not exist" containerID="8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c" Oct 30 18:58:46.303937 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.303857 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c"} err="failed to get container status \"8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c\": rpc error: code = NotFound desc = could not find container \"8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c\": container with ID starting with 8d959a43dd7fb1dc5108820a00aca81c6e66562b84873aa85a3e6e02c5c7976c not found: ID does not exist" Oct 30 18:58:46.308664 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.308632 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:58:46.383426 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.383389 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-kube-api-access-8cxsk\" (UniqueName: \"kubernetes.io/projected/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-kube-api-access-8cxsk\") pod \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\" (UID: \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\") " Oct 30 18:58:46.383620 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.383596 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-plugins-config-volume\") pod \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\" (UID: \"5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9\") " Oct 30 18:58:46.384064 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.384035 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-plugins-config-volume" pod "5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" (UID: "5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:46.384442 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.384356 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:46.386523 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.386492 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-kube-api-access-8cxsk" pod "5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" (UID: "5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9"). InnerVolumeSpecName "kube-api-access-8cxsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:46.485914 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.485829 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9466f2-fd67-4562-b757-8a43a6757a03-kube-api-access-fxqvp\" (UniqueName: \"kubernetes.io/projected/3e9466f2-fd67-4562-b757-8a43a6757a03-kube-api-access-fxqvp\") pod \"3e9466f2-fd67-4562-b757-8a43a6757a03\" (UID: \"3e9466f2-fd67-4562-b757-8a43a6757a03\") " Oct 30 18:58:46.486090 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.485931 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9466f2-fd67-4562-b757-8a43a6757a03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9466f2-fd67-4562-b757-8a43a6757a03-plugins-config-volume\") pod \"3e9466f2-fd67-4562-b757-8a43a6757a03\" (UID: \"3e9466f2-fd67-4562-b757-8a43a6757a03\") " Oct 30 18:58:46.486298 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.486277 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9466f2-fd67-4562-b757-8a43a6757a03-plugins-config-volume" pod "3e9466f2-fd67-4562-b757-8a43a6757a03" (UID: "3e9466f2-fd67-4562-b757-8a43a6757a03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:46.486623 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.486608 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9466f2-fd67-4562-b757-8a43a6757a03-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:46.486671 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.486627 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cxsk\" (UniqueName: \"kubernetes.io/projected/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9-kube-api-access-8cxsk\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:46.488151 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.488126 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9466f2-fd67-4562-b757-8a43a6757a03-kube-api-access-fxqvp" pod "3e9466f2-fd67-4562-b757-8a43a6757a03" (UID: "3e9466f2-fd67-4562-b757-8a43a6757a03"). InnerVolumeSpecName "kube-api-access-fxqvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:46.587826 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.587797 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxqvp\" (UniqueName: \"kubernetes.io/projected/3e9466f2-fd67-4562-b757-8a43a6757a03-kube-api-access-fxqvp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:58:46.626766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.626735 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2"] Oct 30 18:58:46.630759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:46.630734 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0cc6b1-58d46575d-ww7q2"] Oct 30 18:58:47.064759 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:47.064727 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" path="/var/lib/kubelet/pods/5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9/volumes" Oct 30 18:58:47.274760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:47.274679 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" event={"ID":"3e9466f2-fd67-4562-b757-8a43a6757a03","Type":"ContainerDied","Data":"edd0f1f2bff74f76090870f1be1591f370767a6ccc983afe9cf9e920b7124619"} Oct 30 18:58:47.274760 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:47.274745 4862 scope.go:122] "RemoveContainer" containerID="00a4abacc5e63b8b5051c41b8969cd95fc278a5f82d4ad52e897913c4a656768" Oct 30 18:58:47.275302 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:47.274697 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j" Oct 30 18:58:47.327977 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:47.327952 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j"] Oct 30 18:58:47.331949 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:47.331927 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0cc6b1-68b845fc95-6wq4j"] Oct 30 18:58:49.064347 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:49.064317 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9466f2-fd67-4562-b757-8a43a6757a03" path="/var/lib/kubelet/pods/3e9466f2-fd67-4562-b757-8a43a6757a03/volumes" Oct 30 18:58:49.870044 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:58:49.870008 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:58:55.031999 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:55.031967 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k"] Oct 30 18:58:55.043711 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:55.043685 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976b4c8c-6586k"] Oct 30 18:58:55.063648 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:55.063611 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382800df-4182-49aa-89e9-1ac37997127f" path="/var/lib/kubelet/pods/382800df-4182-49aa-89e9-1ac37997127f/volumes" Oct 30 18:58:57.038938 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.038901 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp"] Oct 30 18:58:57.051160 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.051131 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abc57551-6ctmp"] Oct 30 18:58:57.066707 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.066672 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01eb988c-0f90-4346-bca3-af9a971ecd56" path="/var/lib/kubelet/pods/01eb988c-0f90-4346-bca3-af9a971ecd56/volumes" Oct 30 18:58:57.764220 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.764186 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj"] Oct 30 18:58:57.765357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765326 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" containerName="payload-processing" Oct 30 18:58:57.765357 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765350 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" containerName="payload-processing" Oct 30 18:58:57.765564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765368 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3e9466f2-fd67-4562-b757-8a43a6757a03" containerName="payload-pre-processing" Oct 30 18:58:57.765564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765375 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9466f2-fd67-4562-b757-8a43a6757a03" containerName="payload-pre-processing" Oct 30 18:58:57.765564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765423 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ecf115e9-fad0-493f-9d2b-41e28fdd2ee6" containerName="revoke-keys" Oct 30 18:58:57.765564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765433 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf115e9-fad0-493f-9d2b-41e28fdd2ee6" containerName="revoke-keys" Oct 30 18:58:57.765564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765450 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b5fafa9f-cc81-422f-a01f-b381778d26a3" containerName="main" Oct 30 18:58:57.765564 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765456 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fafa9f-cc81-422f-a01f-b381778d26a3" containerName="main" Oct 30 18:58:57.765871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765665 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccf86cb-6638-4e77-a72c-5ba2a0f4e7b9" containerName="payload-processing" Oct 30 18:58:57.765871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765678 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9466f2-fd67-4562-b757-8a43a6757a03" containerName="payload-pre-processing" Oct 30 18:58:57.765871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765698 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf115e9-fad0-493f-9d2b-41e28fdd2ee6" containerName="revoke-keys" Oct 30 18:58:57.765871 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.765708 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fafa9f-cc81-422f-a01f-b381778d26a3" containerName="main" Oct 30 18:58:57.770486 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.770465 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:57.788411 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.788387 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj"] Oct 30 18:58:57.794792 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.794760 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2820d9\"/\"kube-root-ca.crt\"" Oct 30 18:58:57.794936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.794805 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2820d9\"/\"openshift-service-ca.crt\"" Oct 30 18:58:57.797518 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.797497 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2820d9\"/\"test-model-2820d9-kserve-self-signed-certs\"" Oct 30 18:58:57.797628 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.797539 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2820d9\"/\"default-dockercfg-9jpmp\"" Oct 30 18:58:57.908928 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.908893 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-model-cache\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:57.909199 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.909175 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-tmp-dir\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:57.909324 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.909305 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e9e7c8-5abc-448f-8dde-864c9238c181-tls-certs\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:57.909404 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.909383 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-dshm\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:57.909577 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.909557 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gk99\" (UniqueName: \"kubernetes.io/projected/16e9e7c8-5abc-448f-8dde-864c9238c181-kube-api-access-6gk99\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:57.909641 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:57.909625 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-home\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.015438 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.015328 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-tmp-dir\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.015622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.015437 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e9e7c8-5abc-448f-8dde-864c9238c181-tls-certs\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.015622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.015476 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-dshm\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.015622 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.015608 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gk99\" (UniqueName: \"kubernetes.io/projected/16e9e7c8-5abc-448f-8dde-864c9238c181-kube-api-access-6gk99\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.015872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.015665 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-home\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.015872 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.015723 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-model-cache\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.016515 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.016470 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-model-cache\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.016764 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.016745 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-tmp-dir\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.017692 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.017668 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-home\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.030213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.021891 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e9e7c8-5abc-448f-8dde-864c9238c181-tls-certs\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.033289 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.031454 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-dshm\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.042303 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.039823 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gk99\" (UniqueName: \"kubernetes.io/projected/16e9e7c8-5abc-448f-8dde-864c9238c181-kube-api-access-6gk99\") pod \"test-model-2820d9-kserve-6944cb4bcc-728fj\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.097216 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.094555 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:58:58.557279 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:58.554731 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj"] Oct 30 18:58:58.557279 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:58:58.555076 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16e9e7c8_5abc_448f_8dde_864c9238c181.slice/crio-ac655b97351669bf8fbbec6f7b561094f709818f56268bd2049fc8e369b76e8a WatchSource:0}: Error finding container ac655b97351669bf8fbbec6f7b561094f709818f56268bd2049fc8e369b76e8a: Status 404 returned error can't find the container with id ac655b97351669bf8fbbec6f7b561094f709818f56268bd2049fc8e369b76e8a Oct 30 18:58:59.409692 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:59.409661 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" event={"ID":"16e9e7c8-5abc-448f-8dde-864c9238c181","Type":"ContainerStarted","Data":"8007d855310410363d64389f8aecfa5b87604cea8e7e13ed2c8559a6b66d19ef"} Oct 30 18:58:59.410074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:59.409701 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" event={"ID":"16e9e7c8-5abc-448f-8dde-864c9238c181","Type":"ContainerStarted","Data":"ac655b97351669bf8fbbec6f7b561094f709818f56268bd2049fc8e369b76e8a"} Oct 30 18:58:59.449799 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:58:59.449743 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" podStartSLOduration=2.449729403 podStartE2EDuration="2.449729403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:59.448458343 +0000 UTC m=+3638.497503258" watchObservedRunningTime="2026-10-30 18:58:59.449729403 +0000 UTC m=+3638.498774316" Oct 30 18:59:00.095127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:00.095098 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:59:03.037766 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:03.037681 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 18:59:08.030770 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:08.030735 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll"] Oct 30 18:59:08.042680 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:08.042650 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f01aaf7-pw4ll"] Oct 30 18:59:08.095438 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:08.095410 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:59:08.108793 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:08.108768 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:59:08.503667 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:08.503638 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 18:59:09.067066 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:09.067033 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f1b6ee-f340-428c-9287-f0f5630c3640" path="/var/lib/kubelet/pods/a8f1b6ee-f340-428c-9287-f0f5630c3640/volumes" Oct 30 18:59:19.872801 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:59:19.872762 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:59:21.031210 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:21.031177 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj"] Oct 30 18:59:21.034878 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:21.034849 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b673134-kgfbj"] Oct 30 18:59:21.065733 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:21.065704 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5c2a4c-ad82-4dd3-9e5e-820664cc443f" path="/var/lib/kubelet/pods/0c5c2a4c-ad82-4dd3-9e5e-820664cc443f/volumes" Oct 30 18:59:22.773111 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.773067 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk"] Oct 30 18:59:22.779853 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.779832 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:22.809074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.809040 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9f6b7c\"/\"openshift-service-ca.crt\"" Oct 30 18:59:22.809074 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.809036 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9f6b7c\"/\"kube-root-ca.crt\"" Oct 30 18:59:22.809336 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.809143 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9f6b7c\"/\"test-model-9f6b7c-kserve-self-signed-certs\"" Oct 30 18:59:22.809398 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.809336 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9f6b7c\"/\"default-dockercfg-tqpvl\"" Oct 30 18:59:22.813882 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.813856 4862 scope.go:122] "RemoveContainer" containerID="f0d9bb79e5a5ba602404c68ffedcc8ecc17f44509633228c6ff228f6f72cc4d2" Oct 30 18:59:22.820528 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.820501 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk"] Oct 30 18:59:22.830127 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.830106 4862 scope.go:122] "RemoveContainer" containerID="a72aa390b48074d4f5b4de6bceb1d77c64260ae117f504d3641dcd67ecedab7a" Oct 30 18:59:22.942132 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.942092 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-home\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:22.942335 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.942239 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-model-cache\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:22.942419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.942332 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-dshm\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:22.942419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.942376 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfsj2\" (UniqueName: \"kubernetes.io/projected/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-kube-api-access-qfsj2\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:22.942419 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.942401 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tmp-dir\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:22.942596 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:22.942514 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tls-certs\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.047344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.046362 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-home\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.047344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.046504 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-model-cache\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.047344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.046542 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-dshm\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.047344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.046600 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfsj2\" (UniqueName: \"kubernetes.io/projected/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-kube-api-access-qfsj2\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.047344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.046639 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tmp-dir\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.047344 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.046732 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tls-certs\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.059282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.052345 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-home\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.059282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.052600 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-model-cache\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.059282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.053144 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tmp-dir\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.059282 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.054829 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tls-certs\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.080288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.078968 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfsj2\" (UniqueName: \"kubernetes.io/projected/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-kube-api-access-qfsj2\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.104288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.099629 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-dshm\") pod \"test-model-9f6b7c-kserve-6d6d64489c-kgqlk\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.110331 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.109572 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:23.466601 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:59:23.466568 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod893e9668_5fdc_43b6_ab9d_b5e9d9f54410.slice/crio-7ccdf12eadca837d38a513a2bfcfc1b16ed75799b52a5496945b89526798d2ef WatchSource:0}: Error finding container 7ccdf12eadca837d38a513a2bfcfc1b16ed75799b52a5496945b89526798d2ef: Status 404 returned error can't find the container with id 7ccdf12eadca837d38a513a2bfcfc1b16ed75799b52a5496945b89526798d2ef Oct 30 18:59:23.468869 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.467475 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk"] Oct 30 18:59:23.636317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.636111 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" event={"ID":"893e9668-5fdc-43b6-ab9d-b5e9d9f54410","Type":"ContainerStarted","Data":"ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0"} Oct 30 18:59:23.636317 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.636162 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" event={"ID":"893e9668-5fdc-43b6-ab9d-b5e9d9f54410","Type":"ContainerStarted","Data":"7ccdf12eadca837d38a513a2bfcfc1b16ed75799b52a5496945b89526798d2ef"} Oct 30 18:59:23.677777 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:23.677727 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" podStartSLOduration=1.677712335 podStartE2EDuration="1.677712335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:23.676717481 +0000 UTC m=+3662.725762406" watchObservedRunningTime="2026-10-30 18:59:23.677712335 +0000 UTC m=+3662.726757247" Oct 30 18:59:24.113982 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:24.113941 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:28.208234 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:28.208193 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h"] Oct 30 18:59:28.212461 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:28.212434 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0cc6b1-rgb4h"] Oct 30 18:59:29.064697 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:29.064655 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf115e9-fad0-493f-9d2b-41e28fdd2ee6" path="/var/lib/kubelet/pods/ecf115e9-fad0-493f-9d2b-41e28fdd2ee6/volumes" Oct 30 18:59:32.049613 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:32.049575 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn"] Oct 30 18:59:32.057589 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:32.057549 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5622de06-ppjqn"] Oct 30 18:59:33.064161 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:33.064130 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8222f63-efc7-419b-afff-bb79fe5a599e" path="/var/lib/kubelet/pods/b8222f63-efc7-419b-afff-bb79fe5a599e/volumes" Oct 30 18:59:33.113488 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:33.113461 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:33.126213 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:33.126192 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:33.737385 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:33.737352 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 18:59:36.471334 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.468582 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw"] Oct 30 18:59:36.489137 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.489095 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw"] Oct 30 18:59:36.489362 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.489225 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.512214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.504786 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 18:59:36.512214 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.504824 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:59:36.625959 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.625931 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.626173 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.626158 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sts\" (UniqueName: \"kubernetes.io/projected/7854feef-3e42-41d2-b934-ce2f22775b58-kube-api-access-76sts\") pod \"maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.626227 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.626200 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.728239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.728155 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76sts\" (UniqueName: \"kubernetes.io/projected/7854feef-3e42-41d2-b934-ce2f22775b58-kube-api-access-76sts\") pod \"maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.728239 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.728209 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.728541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.728415 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.755329 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.750325 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.757980 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.757923 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.758783 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.758762 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sts\" (UniqueName: \"kubernetes.io/projected/7854feef-3e42-41d2-b934-ce2f22775b58-kube-api-access-76sts\") pod \"maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:36.816762 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:36.816739 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:37.205541 ip-10-0-128-246 kubenswrapper[4862]: W1030 18:59:37.205513 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7854feef_3e42_41d2_b934_ce2f22775b58.slice/crio-7b32e29ce17609e7fa5ded3a129d3333797111bb92715e6858cbd9e06cf21050 WatchSource:0}: Error finding container 7b32e29ce17609e7fa5ded3a129d3333797111bb92715e6858cbd9e06cf21050: Status 404 returned error can't find the container with id 7b32e29ce17609e7fa5ded3a129d3333797111bb92715e6858cbd9e06cf21050 Oct 30 18:59:37.206656 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:37.206598 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw"] Oct 30 18:59:37.803435 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:37.803404 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" event={"ID":"7854feef-3e42-41d2-b934-ce2f22775b58","Type":"ContainerStarted","Data":"def9a9bf51d92f009f6853fa1d86447c565c9e8f978e06409fa2f139b5a224a6"} Oct 30 18:59:37.803775 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:37.803467 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" event={"ID":"7854feef-3e42-41d2-b934-ce2f22775b58","Type":"ContainerStarted","Data":"7b32e29ce17609e7fa5ded3a129d3333797111bb92715e6858cbd9e06cf21050"} Oct 30 18:59:38.821288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:38.817330 4862 generic.go:356] "Generic (PLEG): container finished" podID="7854feef-3e42-41d2-b934-ce2f22775b58" containerID="def9a9bf51d92f009f6853fa1d86447c565c9e8f978e06409fa2f139b5a224a6" exitCode=0 Oct 30 18:59:38.821288 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:38.817634 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" event={"ID":"7854feef-3e42-41d2-b934-ce2f22775b58","Type":"ContainerDied","Data":"def9a9bf51d92f009f6853fa1d86447c565c9e8f978e06409fa2f139b5a224a6"} Oct 30 18:59:40.073050 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.073024 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:40.117558 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.117477 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7854feef-3e42-41d2-b934-ce2f22775b58-kube-api-access-76sts\" (UniqueName: \"kubernetes.io/projected/7854feef-3e42-41d2-b934-ce2f22775b58-kube-api-access-76sts\") pod \"7854feef-3e42-41d2-b934-ce2f22775b58\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " Oct 30 18:59:40.117718 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.117636 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-openshift-ca\") pod \"7854feef-3e42-41d2-b934-ce2f22775b58\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " Oct 30 18:59:40.117936 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.117903 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-xks-ca\") pod \"7854feef-3e42-41d2-b934-ce2f22775b58\" (UID: \"7854feef-3e42-41d2-b934-ce2f22775b58\") " Oct 30 18:59:40.118088 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.118064 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-openshift-ca" pod "7854feef-3e42-41d2-b934-ce2f22775b58" (UID: "7854feef-3e42-41d2-b934-ce2f22775b58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:40.118560 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.118539 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:59:40.119896 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.119875 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7854feef-3e42-41d2-b934-ce2f22775b58-kube-api-access-76sts" pod "7854feef-3e42-41d2-b934-ce2f22775b58" (UID: "7854feef-3e42-41d2-b934-ce2f22775b58"). InnerVolumeSpecName "kube-api-access-76sts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:40.120223 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.120203 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-xks-ca" pod "7854feef-3e42-41d2-b934-ce2f22775b58" (UID: "7854feef-3e42-41d2-b934-ce2f22775b58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:40.219784 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.219752 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7854feef-3e42-41d2-b934-ce2f22775b58-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:59:40.219784 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.219781 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76sts\" (UniqueName: \"kubernetes.io/projected/7854feef-3e42-41d2-b934-ce2f22775b58-kube-api-access-76sts\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 18:59:40.858905 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.858867 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" event={"ID":"7854feef-3e42-41d2-b934-ce2f22775b58","Type":"ContainerDied","Data":"7b32e29ce17609e7fa5ded3a129d3333797111bb92715e6858cbd9e06cf21050"} Oct 30 18:59:40.859068 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.858913 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw" Oct 30 18:59:40.859068 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:40.858956 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b32e29ce17609e7fa5ded3a129d3333797111bb92715e6858cbd9e06cf21050" Oct 30 18:59:45.902529 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:45.902500 4862 generic.go:356] "Generic (PLEG): container finished" podID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerID="860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1" exitCode=0 Oct 30 18:59:45.902888 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:45.902571 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" event={"ID":"a32dd5fc-27df-453f-b026-3811e7fac2f1","Type":"ContainerDied","Data":"860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1"} Oct 30 18:59:45.903006 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:45.902988 4862 scope.go:122] "RemoveContainer" containerID="860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1" Oct 30 18:59:45.905155 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:45.905124 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:59:45.905226 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:45.905198 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:59:45.905283 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:45.905230 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:59:46.900961 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:46.900930 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7"] Oct 30 18:59:46.912529 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:46.912501 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" event={"ID":"a32dd5fc-27df-453f-b026-3811e7fac2f1","Type":"ContainerStarted","Data":"802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2"} Oct 30 18:59:46.959342 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:46.959308 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8"] Oct 30 18:59:46.988858 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:46.988819 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:59:46.989788 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:46.989754 4862 patch_prober.go:29] interesting pod/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:59:46.989942 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:46.989827 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 18:59:47.989541 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:47.989506 4862 patch_prober.go:29] interesting pod/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:59:47.990030 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:47.989564 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 18:59:48.988812 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:48.988774 4862 patch_prober.go:29] interesting pod/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:59:48.988999 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:48.988838 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 18:59:49.066221 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:49.066191 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw"] Oct 30 18:59:49.066662 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:49.066232 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c52ec7-6v4zw"] Oct 30 18:59:49.990294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:49.989021 4862 patch_prober.go:29] interesting pod/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:59:49.990294 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:49.989099 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 18:59:49.996024 ip-10-0-128-246 kubenswrapper[4862]: E1030 18:59:49.995961 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 18:59:50.989078 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:50.989040 4862 patch_prober.go:29] interesting pod/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:59:50.989557 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:50.989121 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 18:59:51.112636 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:51.112600 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7854feef-3e42-41d2-b934-ce2f22775b58" path="/var/lib/kubelet/pods/7854feef-3e42-41d2-b934-ce2f22775b58/volumes" Oct 30 18:59:51.465421 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:51.465393 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:59:51.901497 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:51.901458 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" podUID="6e82685b-9c65-432b-a863-c39edd6950e4" containerName="payload-processing" containerID="cri-o://a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1" gracePeriod=25 Oct 30 18:59:51.960341 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:51.960289 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" podUID="abd590d7-b8b0-4356-9566-c3e62499c87b" containerName="payload-pre-processing" containerID="cri-o://16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11" gracePeriod=25 Oct 30 18:59:51.989371 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:51.989339 4862 patch_prober.go:29] interesting pod/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" start-of-body= Oct 30 18:59:51.989709 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:51.989388 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.232:15021/healthz/ready\": dial tcp 10.133.0.232:15021: connect: connection refused" Oct 30 18:59:52.994460 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:52.994430 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:59:52.995684 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:52.995663 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 18:59:58.691487 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:58.691446 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" podUID="abd590d7-b8b0-4356-9566-c3e62499c87b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:59:58.691995 ip-10-0-128-246 kubenswrapper[4862]: I1030 18:59:58.691449 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" podUID="6e82685b-9c65-432b-a863-c39edd6950e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:00:00.209349 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.209315 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv"] Oct 30 19:00:00.212193 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.211107 4862 cpu_manager.go:414] "containerMap: removing container" podUID="7854feef-3e42-41d2-b934-ce2f22775b58" containerName="revoke-keys" Oct 30 19:00:00.212409 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.212381 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7854feef-3e42-41d2-b934-ce2f22775b58" containerName="revoke-keys" Oct 30 19:00:00.212707 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.212694 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="7854feef-3e42-41d2-b934-ce2f22775b58" containerName="revoke-keys" Oct 30 19:00:00.220717 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.220693 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:00.221818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.221792 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2"] Oct 30 19:00:00.235746 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.235721 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz"] Oct 30 19:00:00.235988 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.235845 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:00.245326 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.245299 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv"] Oct 30 19:00:00.245542 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.245524 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2"] Oct 30 19:00:00.245765 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.245748 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv"] Oct 30 19:00:00.251488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.251464 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:00.251786 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.251763 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d"] Oct 30 19:00:00.252107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.251961 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:00.252777 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.252703 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:00:00.266662 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.266636 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:00.266973 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.266953 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz"] Oct 30 19:00:00.274666 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.274630 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4"] Oct 30 19:00:00.292961 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.292919 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:00.297471 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.297403 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv"] Oct 30 19:00:00.301605 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.301586 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d"] Oct 30 19:00:00.307726 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.306529 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9"] Oct 30 19:00:00.313690 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.313656 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4"] Oct 30 19:00:00.313998 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.313966 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:00.316106 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.316083 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr"] Oct 30 19:00:00.324117 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.323610 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z"] Oct 30 19:00:00.351324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.351295 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:00.353634 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.353603 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d\" (UID: \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:00.354002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.353674 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22br8\" (UniqueName: \"kubernetes.io/projected/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-kube-api-access-22br8\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz\" (UID: \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:00.354096 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.354084 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb79e7b-310f-4013-9d4e-387a6433ba9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4\" (UID: \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:00.354180 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.354164 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmfp9\" (UniqueName: \"kubernetes.io/projected/09356c3c-e94c-4a79-855a-fc550e0eb5f1-kube-api-access-gmfp9\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv\" (UID: \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:00.354563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.354335 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvm46\" (UniqueName: \"kubernetes.io/projected/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-kube-api-access-gvm46\") pod \"maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d\" (UID: \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:00.354563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.354381 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbl68\" (UniqueName: \"kubernetes.io/projected/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-kube-api-access-dbl68\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv\" (UID: \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:00.354563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.354415 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09356c3c-e94c-4a79-855a-fc550e0eb5f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv\" (UID: \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:00.354563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.354553 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8888996a-7141-4f3f-83b3-d87811232f6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2\" (UID: \"8888996a-7141-4f3f-83b3-d87811232f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:00.354920 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.354636 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv\" (UID: \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:00.354920 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.354816 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz\" (UID: \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:00.355036 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.354933 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvd8r\" (UniqueName: \"kubernetes.io/projected/0bb79e7b-310f-4013-9d4e-387a6433ba9f-kube-api-access-hvd8r\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4\" (UID: \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:00.355126 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.355106 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfzpx\" (UniqueName: \"kubernetes.io/projected/8888996a-7141-4f3f-83b3-d87811232f6a-kube-api-access-mfzpx\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2\" (UID: \"8888996a-7141-4f3f-83b3-d87811232f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:00.359793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.359770 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:00.387857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.387166 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9"] Oct 30 19:00:00.405684 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.404155 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w"] Oct 30 19:00:00.425489 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.422250 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:00.425489 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.424204 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr"] Oct 30 19:00:00.434372 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.434344 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj"] Oct 30 19:00:00.445657 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.445635 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z"] Oct 30 19:00:00.445883 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.445865 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w"] Oct 30 19:00:00.446019 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.446007 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7"] Oct 30 19:00:00.446210 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.445784 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:00.452795 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.452778 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:00.453428 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.453412 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp"] Oct 30 19:00:00.457337 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457311 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z\" (UID: \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:00.457416 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457377 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvd8r\" (UniqueName: \"kubernetes.io/projected/0bb79e7b-310f-4013-9d4e-387a6433ba9f-kube-api-access-hvd8r\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4\" (UID: \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:00.457457 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457439 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfzpx\" (UniqueName: \"kubernetes.io/projected/8888996a-7141-4f3f-83b3-d87811232f6a-kube-api-access-mfzpx\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2\" (UID: \"8888996a-7141-4f3f-83b3-d87811232f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:00.457529 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457513 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5496e-2536-4aae-a65e-76e2bac7b327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr\" (UID: \"d9f5496e-2536-4aae-a65e-76e2bac7b327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:00.457575 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457559 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea8704-add1-4406-8aa4-d46f42e68614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9\" (UID: \"b4ea8704-add1-4406-8aa4-d46f42e68614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:00.457626 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457612 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d\" (UID: \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:00.457672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457657 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22br8\" (UniqueName: \"kubernetes.io/projected/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-kube-api-access-22br8\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz\" (UID: \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:00.457759 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457741 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6zn9\" (UniqueName: \"kubernetes.io/projected/b4ea8704-add1-4406-8aa4-d46f42e68614-kube-api-access-j6zn9\") pod \"maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9\" (UID: \"b4ea8704-add1-4406-8aa4-d46f42e68614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:00.457821 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457806 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb79e7b-310f-4013-9d4e-387a6433ba9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4\" (UID: \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:00.457899 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457886 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmfp9\" (UniqueName: \"kubernetes.io/projected/09356c3c-e94c-4a79-855a-fc550e0eb5f1-kube-api-access-gmfp9\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv\" (UID: \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:00.457936 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457916 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbw2p\" (UniqueName: \"kubernetes.io/projected/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-kube-api-access-gbw2p\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z\" (UID: \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:00.457990 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457963 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvm46\" (UniqueName: \"kubernetes.io/projected/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-kube-api-access-gvm46\") pod \"maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d\" (UID: \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:00.458034 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.457991 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbl68\" (UniqueName: \"kubernetes.io/projected/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-kube-api-access-dbl68\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv\" (UID: \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:00.458034 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.458017 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09356c3c-e94c-4a79-855a-fc550e0eb5f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv\" (UID: \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:00.458101 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.458052 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8888996a-7141-4f3f-83b3-d87811232f6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2\" (UID: \"8888996a-7141-4f3f-83b3-d87811232f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:00.458152 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.458137 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv\" (UID: \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:00.458228 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.458213 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srgpp\" (UniqueName: \"kubernetes.io/projected/d9f5496e-2536-4aae-a65e-76e2bac7b327-kube-api-access-srgpp\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr\" (UID: \"d9f5496e-2536-4aae-a65e-76e2bac7b327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:00.459015 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.459001 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:00.459942 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.459904 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d\" (UID: \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:00.460672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.460646 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb79e7b-310f-4013-9d4e-387a6433ba9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4\" (UID: \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:00.460738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.460653 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09356c3c-e94c-4a79-855a-fc550e0eb5f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv\" (UID: \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:00.461606 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.461581 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv\" (UID: \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:00.461685 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.461585 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8888996a-7141-4f3f-83b3-d87811232f6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2\" (UID: \"8888996a-7141-4f3f-83b3-d87811232f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:00.461793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.461772 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz\" (UID: \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:00.463119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.463098 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz\" (UID: \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:00.466186 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.466168 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj"] Oct 30 19:00:00.470933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.470913 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz"] Oct 30 19:00:00.479230 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.479212 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8"] Oct 30 19:00:00.479517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.479504 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:00.484510 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.484484 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:00.484769 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.484752 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp"] Oct 30 19:00:00.489288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.488171 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22br8\" (UniqueName: \"kubernetes.io/projected/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-kube-api-access-22br8\") pod \"maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz\" (UID: \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:00.489288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.488183 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfzpx\" (UniqueName: \"kubernetes.io/projected/8888996a-7141-4f3f-83b3-d87811232f6a-kube-api-access-mfzpx\") pod \"maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2\" (UID: \"8888996a-7141-4f3f-83b3-d87811232f6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:00.489288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.488233 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7"] Oct 30 19:00:00.489563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.489539 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvd8r\" (UniqueName: \"kubernetes.io/projected/0bb79e7b-310f-4013-9d4e-387a6433ba9f-kube-api-access-hvd8r\") pod \"maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4\" (UID: \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:00.489821 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.489801 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvm46\" (UniqueName: \"kubernetes.io/projected/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-kube-api-access-gvm46\") pod \"maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d\" (UID: \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:00.490383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.490359 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbl68\" (UniqueName: \"kubernetes.io/projected/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-kube-api-access-dbl68\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv\" (UID: \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:00.490892 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.490872 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmfp9\" (UniqueName: \"kubernetes.io/projected/09356c3c-e94c-4a79-855a-fc550e0eb5f1-kube-api-access-gmfp9\") pod \"maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv\" (UID: \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:00.497337 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.493029 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7"] Oct 30 19:00:00.498692 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.498676 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:00.499641 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.499623 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz"] Oct 30 19:00:00.505112 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.505087 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk"] Oct 30 19:00:00.512984 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.512959 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8"] Oct 30 19:00:00.513096 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.513064 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:00.516737 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.516708 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7"] Oct 30 19:00:00.523674 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.523657 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk"] Oct 30 19:00:00.563073 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563040 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbw2p\" (UniqueName: \"kubernetes.io/projected/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-kube-api-access-gbw2p\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z\" (UID: \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:00.563210 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563096 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72239397-4285-47ff-b37f-c42943e13b82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8\" (UID: \"72239397-4285-47ff-b37f-c42943e13b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:00.563210 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563129 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf45ce-60c7-44a1-91bf-21236823cbff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w\" (UID: \"cfdf45ce-60c7-44a1-91bf-21236823cbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:00.563210 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563183 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpq9d\" (UniqueName: \"kubernetes.io/projected/ad42ac0a-7ac9-4996-a585-a77c79873164-kube-api-access-xpq9d\") pod \"maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp\" (UID: \"ad42ac0a-7ac9-4996-a585-a77c79873164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:00.563558 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563537 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d313143-8642-4426-8660-66d58f0702b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk\" (UID: \"7d313143-8642-4426-8660-66d58f0702b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:00.563618 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563569 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efeda39-fc25-4232-8bba-e023790536cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7\" (UID: \"1efeda39-fc25-4232-8bba-e023790536cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:00.563618 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563612 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgpp\" (UniqueName: \"kubernetes.io/projected/d9f5496e-2536-4aae-a65e-76e2bac7b327-kube-api-access-srgpp\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr\" (UID: \"d9f5496e-2536-4aae-a65e-76e2bac7b327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:00.563732 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563654 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwknk\" (UniqueName: \"kubernetes.io/projected/7d313143-8642-4426-8660-66d58f0702b3-kube-api-access-dwknk\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk\" (UID: \"7d313143-8642-4426-8660-66d58f0702b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:00.563732 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563676 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwbt\" (UniqueName: \"kubernetes.io/projected/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-kube-api-access-sjwbt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7\" (UID: \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:00.563732 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563700 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5704fe4a-6a39-418d-93ef-c5080c48b9a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj\" (UID: \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:00.563888 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563741 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z\" (UID: \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:00.563888 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563773 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88rtz\" (UniqueName: \"kubernetes.io/projected/38926db9-ad05-4840-86a2-8f066e0de74c-kube-api-access-88rtz\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz\" (UID: \"38926db9-ad05-4840-86a2-8f066e0de74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:00.563888 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563809 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7\" (UID: \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:00.563888 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563837 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxcx\" (UniqueName: \"kubernetes.io/projected/72239397-4285-47ff-b37f-c42943e13b82-kube-api-access-ckxcx\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8\" (UID: \"72239397-4285-47ff-b37f-c42943e13b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:00.564107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563892 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5496e-2536-4aae-a65e-76e2bac7b327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr\" (UID: \"d9f5496e-2536-4aae-a65e-76e2bac7b327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:00.564107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563926 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea8704-add1-4406-8aa4-d46f42e68614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9\" (UID: \"b4ea8704-add1-4406-8aa4-d46f42e68614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:00.564107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563962 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfcrv\" (UniqueName: \"kubernetes.io/projected/cfdf45ce-60c7-44a1-91bf-21236823cbff-kube-api-access-wfcrv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w\" (UID: \"cfdf45ce-60c7-44a1-91bf-21236823cbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:00.564107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.563998 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad42ac0a-7ac9-4996-a585-a77c79873164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp\" (UID: \"ad42ac0a-7ac9-4996-a585-a77c79873164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:00.564107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.564020 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6zn9\" (UniqueName: \"kubernetes.io/projected/b4ea8704-add1-4406-8aa4-d46f42e68614-kube-api-access-j6zn9\") pod \"maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9\" (UID: \"b4ea8704-add1-4406-8aa4-d46f42e68614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:00.564107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.564055 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx9ft\" (UniqueName: \"kubernetes.io/projected/1efeda39-fc25-4232-8bba-e023790536cc-kube-api-access-nx9ft\") pod \"maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7\" (UID: \"1efeda39-fc25-4232-8bba-e023790536cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:00.564107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.564100 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqzr\" (UniqueName: \"kubernetes.io/projected/5704fe4a-6a39-418d-93ef-c5080c48b9a5-kube-api-access-6gqzr\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj\" (UID: \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:00.564498 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.564142 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926db9-ad05-4840-86a2-8f066e0de74c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz\" (UID: \"38926db9-ad05-4840-86a2-8f066e0de74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:00.565146 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.565117 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z\" (UID: \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:00.565464 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.565435 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea8704-add1-4406-8aa4-d46f42e68614-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9\" (UID: \"b4ea8704-add1-4406-8aa4-d46f42e68614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:00.565563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.565440 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5496e-2536-4aae-a65e-76e2bac7b327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr\" (UID: \"d9f5496e-2536-4aae-a65e-76e2bac7b327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:00.583060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.583020 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbw2p\" (UniqueName: \"kubernetes.io/projected/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-kube-api-access-gbw2p\") pod \"maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z\" (UID: \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:00.584434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.584410 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6zn9\" (UniqueName: \"kubernetes.io/projected/b4ea8704-add1-4406-8aa4-d46f42e68614-kube-api-access-j6zn9\") pod \"maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9\" (UID: \"b4ea8704-add1-4406-8aa4-d46f42e68614\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:00.584523 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.584412 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgpp\" (UniqueName: \"kubernetes.io/projected/d9f5496e-2536-4aae-a65e-76e2bac7b327-kube-api-access-srgpp\") pod \"maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr\" (UID: \"d9f5496e-2536-4aae-a65e-76e2bac7b327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:00.588506 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.588486 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:00.592708 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.592690 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:00.608409 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.605201 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:00.632940 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.632641 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:00.648143 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.647464 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:00.677205 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.677169 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7\" (UID: \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:00.677436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.677227 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxcx\" (UniqueName: \"kubernetes.io/projected/72239397-4285-47ff-b37f-c42943e13b82-kube-api-access-ckxcx\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8\" (UID: \"72239397-4285-47ff-b37f-c42943e13b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:00.677436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.677403 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfcrv\" (UniqueName: \"kubernetes.io/projected/cfdf45ce-60c7-44a1-91bf-21236823cbff-kube-api-access-wfcrv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w\" (UID: \"cfdf45ce-60c7-44a1-91bf-21236823cbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:00.677579 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.677463 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad42ac0a-7ac9-4996-a585-a77c79873164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp\" (UID: \"ad42ac0a-7ac9-4996-a585-a77c79873164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:00.677579 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.677519 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx9ft\" (UniqueName: \"kubernetes.io/projected/1efeda39-fc25-4232-8bba-e023790536cc-kube-api-access-nx9ft\") pod \"maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7\" (UID: \"1efeda39-fc25-4232-8bba-e023790536cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:00.677697 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.677584 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqzr\" (UniqueName: \"kubernetes.io/projected/5704fe4a-6a39-418d-93ef-c5080c48b9a5-kube-api-access-6gqzr\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj\" (UID: \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:00.679375 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.679314 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926db9-ad05-4840-86a2-8f066e0de74c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz\" (UID: \"38926db9-ad05-4840-86a2-8f066e0de74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.681684 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72239397-4285-47ff-b37f-c42943e13b82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8\" (UID: \"72239397-4285-47ff-b37f-c42943e13b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.681733 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf45ce-60c7-44a1-91bf-21236823cbff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w\" (UID: \"cfdf45ce-60c7-44a1-91bf-21236823cbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.681801 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpq9d\" (UniqueName: \"kubernetes.io/projected/ad42ac0a-7ac9-4996-a585-a77c79873164-kube-api-access-xpq9d\") pod \"maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp\" (UID: \"ad42ac0a-7ac9-4996-a585-a77c79873164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.681921 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d313143-8642-4426-8660-66d58f0702b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk\" (UID: \"7d313143-8642-4426-8660-66d58f0702b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.681955 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efeda39-fc25-4232-8bba-e023790536cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7\" (UID: \"1efeda39-fc25-4232-8bba-e023790536cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.682063 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwknk\" (UniqueName: \"kubernetes.io/projected/7d313143-8642-4426-8660-66d58f0702b3-kube-api-access-dwknk\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk\" (UID: \"7d313143-8642-4426-8660-66d58f0702b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.682092 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwbt\" (UniqueName: \"kubernetes.io/projected/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-kube-api-access-sjwbt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7\" (UID: \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.682122 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5704fe4a-6a39-418d-93ef-c5080c48b9a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj\" (UID: \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.682176 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.682231 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88rtz\" (UniqueName: \"kubernetes.io/projected/38926db9-ad05-4840-86a2-8f066e0de74c-kube-api-access-88rtz\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz\" (UID: \"38926db9-ad05-4840-86a2-8f066e0de74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.682754 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf45ce-60c7-44a1-91bf-21236823cbff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w\" (UID: \"cfdf45ce-60c7-44a1-91bf-21236823cbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.683092 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad42ac0a-7ac9-4996-a585-a77c79873164-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp\" (UID: \"ad42ac0a-7ac9-4996-a585-a77c79873164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.683803 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5704fe4a-6a39-418d-93ef-c5080c48b9a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj\" (UID: \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.683806 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926db9-ad05-4840-86a2-8f066e0de74c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz\" (UID: \"38926db9-ad05-4840-86a2-8f066e0de74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:00.684130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.683809 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d313143-8642-4426-8660-66d58f0702b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk\" (UID: \"7d313143-8642-4426-8660-66d58f0702b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:00.685030 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.684647 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7\" (UID: \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:00.685030 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.684725 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efeda39-fc25-4232-8bba-e023790536cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7\" (UID: \"1efeda39-fc25-4232-8bba-e023790536cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:00.685214 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.685140 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72239397-4285-47ff-b37f-c42943e13b82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8\" (UID: \"72239397-4285-47ff-b37f-c42943e13b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:00.693439 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.690569 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:00.717354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.717034 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwknk\" (UniqueName: \"kubernetes.io/projected/7d313143-8642-4426-8660-66d58f0702b3-kube-api-access-dwknk\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk\" (UID: \"7d313143-8642-4426-8660-66d58f0702b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:00.717769 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.717623 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:00.721827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.721245 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88rtz\" (UniqueName: \"kubernetes.io/projected/38926db9-ad05-4840-86a2-8f066e0de74c-kube-api-access-88rtz\") pod \"maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz\" (UID: \"38926db9-ad05-4840-86a2-8f066e0de74c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:00.721827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.721468 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:00.725905 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.725646 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx9ft\" (UniqueName: \"kubernetes.io/projected/1efeda39-fc25-4232-8bba-e023790536cc-kube-api-access-nx9ft\") pod \"maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7\" (UID: \"1efeda39-fc25-4232-8bba-e023790536cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:00.726354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.726324 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfcrv\" (UniqueName: \"kubernetes.io/projected/cfdf45ce-60c7-44a1-91bf-21236823cbff-kube-api-access-wfcrv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w\" (UID: \"cfdf45ce-60c7-44a1-91bf-21236823cbff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:00.728029 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.728007 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxcx\" (UniqueName: \"kubernetes.io/projected/72239397-4285-47ff-b37f-c42943e13b82-kube-api-access-ckxcx\") pod \"maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8\" (UID: \"72239397-4285-47ff-b37f-c42943e13b82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:00.728214 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.728147 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqzr\" (UniqueName: \"kubernetes.io/projected/5704fe4a-6a39-418d-93ef-c5080c48b9a5-kube-api-access-6gqzr\") pod \"maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj\" (UID: \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:00.731893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.731845 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpq9d\" (UniqueName: \"kubernetes.io/projected/ad42ac0a-7ac9-4996-a585-a77c79873164-kube-api-access-xpq9d\") pod \"maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp\" (UID: \"ad42ac0a-7ac9-4996-a585-a77c79873164\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:00.738492 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.738442 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwbt\" (UniqueName: \"kubernetes.io/projected/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-kube-api-access-sjwbt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7\" (UID: \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:00.768656 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.768406 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:00.777513 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.775760 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:00.796713 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.794657 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:00.840359 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.840228 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:00.847227 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.847075 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:00.868096 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.866795 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:00.873038 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.872995 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:00.881646 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:00.881464 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:01.323529 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.323473 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv"] Oct 30 19:00:01.324287 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.324229 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-4fa3ef7630c3bd47d0b07e615cdb7bc0722748da6acd5d6fc9b64a7c7964c86f WatchSource:0}: Error finding container 4fa3ef7630c3bd47d0b07e615cdb7bc0722748da6acd5d6fc9b64a7c7964c86f: Status 404 returned error can't find the container with id 4fa3ef7630c3bd47d0b07e615cdb7bc0722748da6acd5d6fc9b64a7c7964c86f Oct 30 19:00:01.349549 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.349491 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2"] Oct 30 19:00:01.354208 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.354177 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8888996a_7141_4f3f_83b3_d87811232f6a.slice/crio-73f8ceb7a9b9c5dfa06462e9e5a0ac70c08a098d3c1978347b11511f58ee2cba WatchSource:0}: Error finding container 73f8ceb7a9b9c5dfa06462e9e5a0ac70c08a098d3c1978347b11511f58ee2cba: Status 404 returned error can't find the container with id 73f8ceb7a9b9c5dfa06462e9e5a0ac70c08a098d3c1978347b11511f58ee2cba Oct 30 19:00:01.354552 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.354529 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz"] Oct 30 19:00:01.355015 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.354973 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edcb24a_8fa2_4038_a6a6_778febfd0a7d.slice/crio-6350d55ca46f1fc407ab38d0b6bbf2c9063739c996d7e319a7dfc3bc6bf34b19 WatchSource:0}: Error finding container 6350d55ca46f1fc407ab38d0b6bbf2c9063739c996d7e319a7dfc3bc6bf34b19: Status 404 returned error can't find the container with id 6350d55ca46f1fc407ab38d0b6bbf2c9063739c996d7e319a7dfc3bc6bf34b19 Oct 30 19:00:01.413087 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.412984 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv"] Oct 30 19:00:01.420038 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.420011 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d"] Oct 30 19:00:01.522853 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.522828 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9"] Oct 30 19:00:01.533484 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.533455 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr"] Oct 30 19:00:01.608865 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.608836 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w"] Oct 30 19:00:01.611415 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.611378 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfdf45ce_60c7_44a1_91bf_21236823cbff.slice/crio-5aeeb96764ab8c6ef44390d9d81c64b03c776505735e4a4e0128a3bb21e62fae WatchSource:0}: Error finding container 5aeeb96764ab8c6ef44390d9d81c64b03c776505735e4a4e0128a3bb21e62fae: Status 404 returned error can't find the container with id 5aeeb96764ab8c6ef44390d9d81c64b03c776505735e4a4e0128a3bb21e62fae Oct 30 19:00:01.632381 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.632354 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7"] Oct 30 19:00:01.640821 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.640796 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z"] Oct 30 19:00:01.641537 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.641321 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6fae04_3d28_4cf2_a8d5_a58c9be23015.slice/crio-876a6b5d2e39ef03b6bd60fb25ec7b569af2303de9569298d28d25405d742b9f WatchSource:0}: Error finding container 876a6b5d2e39ef03b6bd60fb25ec7b569af2303de9569298d28d25405d742b9f: Status 404 returned error can't find the container with id 876a6b5d2e39ef03b6bd60fb25ec7b569af2303de9569298d28d25405d742b9f Oct 30 19:00:01.642857 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.642779 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38926db9_ad05_4840_86a2_8f066e0de74c.slice/crio-ab9dd2ca9f09888441de2850f60689cf803b6eda89dab66d554ae7df0a478b29 WatchSource:0}: Error finding container ab9dd2ca9f09888441de2850f60689cf803b6eda89dab66d554ae7df0a478b29: Status 404 returned error can't find the container with id ab9dd2ca9f09888441de2850f60689cf803b6eda89dab66d554ae7df0a478b29 Oct 30 19:00:01.648611 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.648537 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz"] Oct 30 19:00:01.673003 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.672969 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4"] Oct 30 19:00:01.673277 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.673240 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb79e7b_310f_4013_9d4e_387a6433ba9f.slice/crio-d3f9c4e55fe8938b82d41b66b37916129b596d6e6a1c334cf7fe07e21aac5b30 WatchSource:0}: Error finding container d3f9c4e55fe8938b82d41b66b37916129b596d6e6a1c334cf7fe07e21aac5b30: Status 404 returned error can't find the container with id d3f9c4e55fe8938b82d41b66b37916129b596d6e6a1c334cf7fe07e21aac5b30 Oct 30 19:00:01.686160 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.686127 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj"] Oct 30 19:00:01.690111 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.690085 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5704fe4a_6a39_418d_93ef_c5080c48b9a5.slice/crio-5d767820c33c2d5a68df5b13693bf8ff6059443bb4a1fc81302c03b592523e9b WatchSource:0}: Error finding container 5d767820c33c2d5a68df5b13693bf8ff6059443bb4a1fc81302c03b592523e9b: Status 404 returned error can't find the container with id 5d767820c33c2d5a68df5b13693bf8ff6059443bb4a1fc81302c03b592523e9b Oct 30 19:00:01.691724 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.691694 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7"] Oct 30 19:00:01.692585 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.692559 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode365121d_8f28_44c0_8b9c_3ce63f3b8fe8.slice/crio-26f060098b9431555df4a5d0a082bc032637e31b9c5bcc0553eec4f31169b234 WatchSource:0}: Error finding container 26f060098b9431555df4a5d0a082bc032637e31b9c5bcc0553eec4f31169b234: Status 404 returned error can't find the container with id 26f060098b9431555df4a5d0a082bc032637e31b9c5bcc0553eec4f31169b234 Oct 30 19:00:01.759522 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.759498 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp"] Oct 30 19:00:01.760129 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.760099 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad42ac0a_7ac9_4996_a585_a77c79873164.slice/crio-5dcff021e6b62073f5db9dc7dd21999f1294bf72dc538e6a443483b5af9029b7 WatchSource:0}: Error finding container 5dcff021e6b62073f5db9dc7dd21999f1294bf72dc538e6a443483b5af9029b7: Status 404 returned error can't find the container with id 5dcff021e6b62073f5db9dc7dd21999f1294bf72dc538e6a443483b5af9029b7 Oct 30 19:00:01.781698 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.781671 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8"] Oct 30 19:00:01.798745 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:01.798722 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk"] Oct 30 19:00:01.837881 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:00:01.837840 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72239397_4285_47ff_b37f_c42943e13b82.slice/crio-3eb777518c252dcb09aafcd1895a316173db42f74a21d7e85fd1a5652f4c40ba WatchSource:0}: Error finding container 3eb777518c252dcb09aafcd1895a316173db42f74a21d7e85fd1a5652f4c40ba: Status 404 returned error can't find the container with id 3eb777518c252dcb09aafcd1895a316173db42f74a21d7e85fd1a5652f4c40ba Oct 30 19:00:02.205682 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.205652 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" event={"ID":"d9f5496e-2536-4aae-a65e-76e2bac7b327","Type":"ContainerStarted","Data":"6717265fba9c8d7f454cbfa7c8c34ae7873f396e64b01b39a5b6cfd4905c0ad3"} Oct 30 19:00:02.206985 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.206959 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" event={"ID":"8888996a-7141-4f3f-83b3-d87811232f6a","Type":"ContainerStarted","Data":"73f8ceb7a9b9c5dfa06462e9e5a0ac70c08a098d3c1978347b11511f58ee2cba"} Oct 30 19:00:02.208778 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.208752 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" event={"ID":"5edcb24a-8fa2-4038-a6a6-778febfd0a7d","Type":"ContainerStarted","Data":"5a0746e5900e0e3935d8d836ae01d210623d14b29f0c2ce0e670a5a82711df48"} Oct 30 19:00:02.208891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.208800 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" event={"ID":"5edcb24a-8fa2-4038-a6a6-778febfd0a7d","Type":"ContainerStarted","Data":"6350d55ca46f1fc407ab38d0b6bbf2c9063739c996d7e319a7dfc3bc6bf34b19"} Oct 30 19:00:02.210146 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.210125 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" event={"ID":"7d313143-8642-4426-8660-66d58f0702b3","Type":"ContainerStarted","Data":"8ef1c733c58abe8929cd2ddd1dfcfcc01ddd16338753cfff816daef67bf91257"} Oct 30 19:00:02.211465 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.211442 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" event={"ID":"cfdf45ce-60c7-44a1-91bf-21236823cbff","Type":"ContainerStarted","Data":"5aeeb96764ab8c6ef44390d9d81c64b03c776505735e4a4e0128a3bb21e62fae"} Oct 30 19:00:02.212668 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.212648 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" event={"ID":"1d6fae04-3d28-4cf2-a8d5-a58c9be23015","Type":"ContainerStarted","Data":"876a6b5d2e39ef03b6bd60fb25ec7b569af2303de9569298d28d25405d742b9f"} Oct 30 19:00:02.213977 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.213956 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" event={"ID":"8a8916eb-d97c-414b-ad8c-03dfda5aebc6","Type":"ContainerStarted","Data":"4552aa6a07ca815e4b8b4abfa5090cde030c1165a0f0aadaee0d18a049d788f5"} Oct 30 19:00:02.215183 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.215164 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" event={"ID":"0bb79e7b-310f-4013-9d4e-387a6433ba9f","Type":"ContainerStarted","Data":"d3f9c4e55fe8938b82d41b66b37916129b596d6e6a1c334cf7fe07e21aac5b30"} Oct 30 19:00:02.216494 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.216476 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" event={"ID":"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7","Type":"ContainerStarted","Data":"bbf3ab409f764722298c508d6cfe588c751572433a68d4a1ed8c23277cd7656f"} Oct 30 19:00:02.218076 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.218051 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" event={"ID":"1efeda39-fc25-4232-8bba-e023790536cc","Type":"ContainerStarted","Data":"726e37372eb64f74a2c903809b9c0ed9877971f5cdeaec53c63d8fc46de3048c"} Oct 30 19:00:02.219455 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.219434 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" event={"ID":"5704fe4a-6a39-418d-93ef-c5080c48b9a5","Type":"ContainerStarted","Data":"5d767820c33c2d5a68df5b13693bf8ff6059443bb4a1fc81302c03b592523e9b"} Oct 30 19:00:02.220670 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.220648 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" event={"ID":"09356c3c-e94c-4a79-855a-fc550e0eb5f1","Type":"ContainerStarted","Data":"4fa3ef7630c3bd47d0b07e615cdb7bc0722748da6acd5d6fc9b64a7c7964c86f"} Oct 30 19:00:02.221890 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.221868 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" event={"ID":"ad42ac0a-7ac9-4996-a585-a77c79873164","Type":"ContainerStarted","Data":"5dcff021e6b62073f5db9dc7dd21999f1294bf72dc538e6a443483b5af9029b7"} Oct 30 19:00:02.223125 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.223105 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" event={"ID":"b4ea8704-add1-4406-8aa4-d46f42e68614","Type":"ContainerStarted","Data":"90a9d881bbadf9d4086d227ded2d78985738efe09798603bd7aa48c107ff300f"} Oct 30 19:00:02.224379 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.224351 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" event={"ID":"38926db9-ad05-4840-86a2-8f066e0de74c","Type":"ContainerStarted","Data":"ab9dd2ca9f09888441de2850f60689cf803b6eda89dab66d554ae7df0a478b29"} Oct 30 19:00:02.225547 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.225524 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" event={"ID":"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8","Type":"ContainerStarted","Data":"26f060098b9431555df4a5d0a082bc032637e31b9c5bcc0553eec4f31169b234"} Oct 30 19:00:02.226887 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.226867 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" event={"ID":"72239397-4285-47ff-b37f-c42943e13b82","Type":"ContainerStarted","Data":"3eb777518c252dcb09aafcd1895a316173db42f74a21d7e85fd1a5652f4c40ba"} Oct 30 19:00:02.251622 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:02.251561 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" podStartSLOduration=2.251547825 podStartE2EDuration="2.251547825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:02.248295735 +0000 UTC m=+3701.297340641" watchObservedRunningTime="2026-10-30 19:00:02.251547825 +0000 UTC m=+3701.300592738" Oct 30 19:00:03.253768 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.253726 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" event={"ID":"72239397-4285-47ff-b37f-c42943e13b82","Type":"ContainerStarted","Data":"c7c8817c602a0b394df7ec74c018a3acf88d5109f00dd1503d4b3470a39d5915"} Oct 30 19:00:03.258063 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.258032 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" event={"ID":"d9f5496e-2536-4aae-a65e-76e2bac7b327","Type":"ContainerStarted","Data":"c443979759772e70b3eda2de2e4a1a23f59aa5c687c495cce273aa9bdb9f0873"} Oct 30 19:00:03.258357 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.258333 4862 scope.go:122] "RemoveContainer" containerID="c443979759772e70b3eda2de2e4a1a23f59aa5c687c495cce273aa9bdb9f0873" Oct 30 19:00:03.260502 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.260475 4862 generic.go:356] "Generic (PLEG): container finished" podID="8888996a-7141-4f3f-83b3-d87811232f6a" containerID="ad17b2f96d89fc34939452b758b6d3bbce3639b6749ef86cf9b193105db0646d" exitCode=60 Oct 30 19:00:03.260597 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.260582 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" event={"ID":"8888996a-7141-4f3f-83b3-d87811232f6a","Type":"ContainerDied","Data":"ad17b2f96d89fc34939452b758b6d3bbce3639b6749ef86cf9b193105db0646d"} Oct 30 19:00:03.260848 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.260808 4862 scope.go:122] "RemoveContainer" containerID="ad17b2f96d89fc34939452b758b6d3bbce3639b6749ef86cf9b193105db0646d" Oct 30 19:00:03.263201 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.263175 4862 generic.go:356] "Generic (PLEG): container finished" podID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerID="5a0746e5900e0e3935d8d836ae01d210623d14b29f0c2ce0e670a5a82711df48" exitCode=60 Oct 30 19:00:03.263341 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.263318 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" event={"ID":"5edcb24a-8fa2-4038-a6a6-778febfd0a7d","Type":"ContainerDied","Data":"5a0746e5900e0e3935d8d836ae01d210623d14b29f0c2ce0e670a5a82711df48"} Oct 30 19:00:03.263693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.263667 4862 scope.go:122] "RemoveContainer" containerID="5a0746e5900e0e3935d8d836ae01d210623d14b29f0c2ce0e670a5a82711df48" Oct 30 19:00:03.266383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.266317 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" event={"ID":"cfdf45ce-60c7-44a1-91bf-21236823cbff","Type":"ContainerStarted","Data":"0a12037dbb8b0ee25915471900d8cbafbd9ced0c825e9ab688de205c5b2bc88c"} Oct 30 19:00:03.266577 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.266558 4862 scope.go:122] "RemoveContainer" containerID="0a12037dbb8b0ee25915471900d8cbafbd9ced0c825e9ab688de205c5b2bc88c" Oct 30 19:00:03.268093 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.268073 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" event={"ID":"1d6fae04-3d28-4cf2-a8d5-a58c9be23015","Type":"ContainerStarted","Data":"01f002a1963de1c842318736a789293bc5aea59b6720720ab1d38a5d742650ea"} Oct 30 19:00:03.269884 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.269863 4862 generic.go:356] "Generic (PLEG): container finished" podID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerID="7b1dcd73017e8f48be905f9729fff821f203d895c6c202ddec0069b72b6e2814" exitCode=60 Oct 30 19:00:03.269996 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.269940 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" event={"ID":"8a8916eb-d97c-414b-ad8c-03dfda5aebc6","Type":"ContainerDied","Data":"7b1dcd73017e8f48be905f9729fff821f203d895c6c202ddec0069b72b6e2814"} Oct 30 19:00:03.270227 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.270154 4862 scope.go:122] "RemoveContainer" containerID="7b1dcd73017e8f48be905f9729fff821f203d895c6c202ddec0069b72b6e2814" Oct 30 19:00:03.279653 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.279611 4862 generic.go:356] "Generic (PLEG): container finished" podID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerID="b1491fb20fd6b111a9c5c91a2d57f382544d7399c639b316031d0491c6bc4b76" exitCode=60 Oct 30 19:00:03.279788 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.279683 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" event={"ID":"0bb79e7b-310f-4013-9d4e-387a6433ba9f","Type":"ContainerDied","Data":"b1491fb20fd6b111a9c5c91a2d57f382544d7399c639b316031d0491c6bc4b76"} Oct 30 19:00:03.280015 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.279995 4862 scope.go:122] "RemoveContainer" containerID="b1491fb20fd6b111a9c5c91a2d57f382544d7399c639b316031d0491c6bc4b76" Oct 30 19:00:03.290247 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.290132 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" event={"ID":"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7","Type":"ContainerStarted","Data":"af684e2c1436cd6d1b9167b9df67c73f4ff13d1291180b93b7e3831822dd8978"} Oct 30 19:00:03.291946 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.291925 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" event={"ID":"1efeda39-fc25-4232-8bba-e023790536cc","Type":"ContainerStarted","Data":"edea5970407918b20f90a73aa69c9fe7d866cf73972365c673065725bb47b109"} Oct 30 19:00:03.307921 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.307729 4862 generic.go:356] "Generic (PLEG): container finished" podID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerID="3afa152a0d2d16560941a441067296356371c084e8896fc386661d8dbdf659d2" exitCode=60 Oct 30 19:00:03.307921 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.307884 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" event={"ID":"5704fe4a-6a39-418d-93ef-c5080c48b9a5","Type":"ContainerDied","Data":"3afa152a0d2d16560941a441067296356371c084e8896fc386661d8dbdf659d2"} Oct 30 19:00:03.308112 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.308070 4862 scope.go:122] "RemoveContainer" containerID="3afa152a0d2d16560941a441067296356371c084e8896fc386661d8dbdf659d2" Oct 30 19:00:03.311719 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.310878 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" podStartSLOduration=3.310862236 podStartE2EDuration="3.310862236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.303287842 +0000 UTC m=+3702.352332753" watchObservedRunningTime="2026-10-30 19:00:03.310862236 +0000 UTC m=+3702.359907146" Oct 30 19:00:03.311975 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.311730 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" event={"ID":"09356c3c-e94c-4a79-855a-fc550e0eb5f1","Type":"ContainerStarted","Data":"b780c69b8828f2a127abe5590295a5fee76e1ba4ea4c343c84e04f7c42e5de8d"} Oct 30 19:00:03.313982 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.313957 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" event={"ID":"ad42ac0a-7ac9-4996-a585-a77c79873164","Type":"ContainerStarted","Data":"3848278eafc07d64524606439af116f6dc08b98001d5798c85e1d0e6857afeb7"} Oct 30 19:00:03.317114 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.316584 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" event={"ID":"b4ea8704-add1-4406-8aa4-d46f42e68614","Type":"ContainerStarted","Data":"1e26877cb29979861360cc23f8984db8deab73e97497243fd79986762799c028"} Oct 30 19:00:03.318758 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.318547 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" event={"ID":"38926db9-ad05-4840-86a2-8f066e0de74c","Type":"ContainerStarted","Data":"dae406a0a961a97910c126d142274a7c41a6eef4650b290e79a87126b58974d8"} Oct 30 19:00:03.321315 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.321052 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" event={"ID":"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8","Type":"ContainerStarted","Data":"784496bd98f483b174455d31d5102598e92618f42f6bc83ad38f4fc4a0638cea"} Oct 30 19:00:03.477034 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.476982 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" podStartSLOduration=3.476967038 podStartE2EDuration="3.476967038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.430916205 +0000 UTC m=+3702.479961118" watchObservedRunningTime="2026-10-30 19:00:03.476967038 +0000 UTC m=+3702.526011951" Oct 30 19:00:03.594626 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.594574 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" podStartSLOduration=3.59455908 podStartE2EDuration="3.59455908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.591340426 +0000 UTC m=+3702.640385340" watchObservedRunningTime="2026-10-30 19:00:03.59455908 +0000 UTC m=+3702.643604025" Oct 30 19:00:03.632536 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.632477 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" podStartSLOduration=3.6324591760000002 podStartE2EDuration="3.632459176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.630528245 +0000 UTC m=+3702.679573158" watchObservedRunningTime="2026-10-30 19:00:03.632459176 +0000 UTC m=+3702.681504083" Oct 30 19:00:03.787012 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.786963 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" podStartSLOduration=3.786948338 podStartE2EDuration="3.786948338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.782970302 +0000 UTC m=+3702.832015217" watchObservedRunningTime="2026-10-30 19:00:03.786948338 +0000 UTC m=+3702.835993256" Oct 30 19:00:03.847572 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.847530 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" podStartSLOduration=3.847517622 podStartE2EDuration="3.847517622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.845423848 +0000 UTC m=+3702.894468762" watchObservedRunningTime="2026-10-30 19:00:03.847517622 +0000 UTC m=+3702.896562534" Oct 30 19:00:03.893468 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.893427 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" podStartSLOduration=3.893409043 podStartE2EDuration="3.893409043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.893202204 +0000 UTC m=+3702.942247128" watchObservedRunningTime="2026-10-30 19:00:03.893409043 +0000 UTC m=+3702.942453955" Oct 30 19:00:03.934095 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.934051 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" podStartSLOduration=3.93403848 podStartE2EDuration="3.93403848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.932154448 +0000 UTC m=+3702.981199361" watchObservedRunningTime="2026-10-30 19:00:03.93403848 +0000 UTC m=+3702.983083393" Oct 30 19:00:03.978444 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:03.978394 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" podStartSLOduration=3.978377844 podStartE2EDuration="3.978377844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.97335957 +0000 UTC m=+3703.022404483" watchObservedRunningTime="2026-10-30 19:00:03.978377844 +0000 UTC m=+3703.027422758" Oct 30 19:00:04.342592 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.339998 4862 generic.go:356] "Generic (PLEG): container finished" podID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerID="b780c69b8828f2a127abe5590295a5fee76e1ba4ea4c343c84e04f7c42e5de8d" exitCode=60 Oct 30 19:00:04.342592 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.340156 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" event={"ID":"09356c3c-e94c-4a79-855a-fc550e0eb5f1","Type":"ContainerDied","Data":"b780c69b8828f2a127abe5590295a5fee76e1ba4ea4c343c84e04f7c42e5de8d"} Oct 30 19:00:04.342592 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.340785 4862 scope.go:122] "RemoveContainer" containerID="b780c69b8828f2a127abe5590295a5fee76e1ba4ea4c343c84e04f7c42e5de8d" Oct 30 19:00:04.345190 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.345168 4862 generic.go:356] "Generic (PLEG): container finished" podID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerID="3848278eafc07d64524606439af116f6dc08b98001d5798c85e1d0e6857afeb7" exitCode=60 Oct 30 19:00:04.345473 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.345219 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" event={"ID":"ad42ac0a-7ac9-4996-a585-a77c79873164","Type":"ContainerDied","Data":"3848278eafc07d64524606439af116f6dc08b98001d5798c85e1d0e6857afeb7"} Oct 30 19:00:04.345570 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.345511 4862 scope.go:122] "RemoveContainer" containerID="3848278eafc07d64524606439af116f6dc08b98001d5798c85e1d0e6857afeb7" Oct 30 19:00:04.358612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.351886 4862 generic.go:356] "Generic (PLEG): container finished" podID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerID="1e26877cb29979861360cc23f8984db8deab73e97497243fd79986762799c028" exitCode=60 Oct 30 19:00:04.358612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.352015 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" event={"ID":"b4ea8704-add1-4406-8aa4-d46f42e68614","Type":"ContainerDied","Data":"1e26877cb29979861360cc23f8984db8deab73e97497243fd79986762799c028"} Oct 30 19:00:04.358612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.352205 4862 scope.go:122] "RemoveContainer" containerID="1e26877cb29979861360cc23f8984db8deab73e97497243fd79986762799c028" Oct 30 19:00:04.358612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.355328 4862 generic.go:356] "Generic (PLEG): container finished" podID="38926db9-ad05-4840-86a2-8f066e0de74c" containerID="dae406a0a961a97910c126d142274a7c41a6eef4650b290e79a87126b58974d8" exitCode=60 Oct 30 19:00:04.358612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.355406 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" event={"ID":"38926db9-ad05-4840-86a2-8f066e0de74c","Type":"ContainerDied","Data":"dae406a0a961a97910c126d142274a7c41a6eef4650b290e79a87126b58974d8"} Oct 30 19:00:04.358612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.355761 4862 scope.go:122] "RemoveContainer" containerID="dae406a0a961a97910c126d142274a7c41a6eef4650b290e79a87126b58974d8" Oct 30 19:00:04.366875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.363560 4862 generic.go:356] "Generic (PLEG): container finished" podID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerID="784496bd98f483b174455d31d5102598e92618f42f6bc83ad38f4fc4a0638cea" exitCode=60 Oct 30 19:00:04.366875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.363722 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" event={"ID":"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8","Type":"ContainerDied","Data":"784496bd98f483b174455d31d5102598e92618f42f6bc83ad38f4fc4a0638cea"} Oct 30 19:00:04.366875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.364130 4862 scope.go:122] "RemoveContainer" containerID="784496bd98f483b174455d31d5102598e92618f42f6bc83ad38f4fc4a0638cea" Oct 30 19:00:04.373450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.371970 4862 generic.go:356] "Generic (PLEG): container finished" podID="72239397-4285-47ff-b37f-c42943e13b82" containerID="c7c8817c602a0b394df7ec74c018a3acf88d5109f00dd1503d4b3470a39d5915" exitCode=60 Oct 30 19:00:04.373450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.372081 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" event={"ID":"72239397-4285-47ff-b37f-c42943e13b82","Type":"ContainerDied","Data":"c7c8817c602a0b394df7ec74c018a3acf88d5109f00dd1503d4b3470a39d5915"} Oct 30 19:00:04.373450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.372458 4862 scope.go:122] "RemoveContainer" containerID="c7c8817c602a0b394df7ec74c018a3acf88d5109f00dd1503d4b3470a39d5915" Oct 30 19:00:04.374895 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.374874 4862 generic.go:356] "Generic (PLEG): container finished" podID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerID="c443979759772e70b3eda2de2e4a1a23f59aa5c687c495cce273aa9bdb9f0873" exitCode=60 Oct 30 19:00:04.375038 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.375021 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" event={"ID":"d9f5496e-2536-4aae-a65e-76e2bac7b327","Type":"ContainerDied","Data":"c443979759772e70b3eda2de2e4a1a23f59aa5c687c495cce273aa9bdb9f0873"} Oct 30 19:00:04.378568 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.378524 4862 generic.go:356] "Generic (PLEG): container finished" podID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerID="0a12037dbb8b0ee25915471900d8cbafbd9ced0c825e9ab688de205c5b2bc88c" exitCode=60 Oct 30 19:00:04.378568 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.378555 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" event={"ID":"cfdf45ce-60c7-44a1-91bf-21236823cbff","Type":"ContainerDied","Data":"0a12037dbb8b0ee25915471900d8cbafbd9ced0c825e9ab688de205c5b2bc88c"} Oct 30 19:00:04.380692 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.380589 4862 generic.go:356] "Generic (PLEG): container finished" podID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerID="01f002a1963de1c842318736a789293bc5aea59b6720720ab1d38a5d742650ea" exitCode=60 Oct 30 19:00:04.380692 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.380652 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" event={"ID":"1d6fae04-3d28-4cf2-a8d5-a58c9be23015","Type":"ContainerDied","Data":"01f002a1963de1c842318736a789293bc5aea59b6720720ab1d38a5d742650ea"} Oct 30 19:00:04.384885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.381660 4862 scope.go:122] "RemoveContainer" containerID="01f002a1963de1c842318736a789293bc5aea59b6720720ab1d38a5d742650ea" Oct 30 19:00:04.384885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.384112 4862 generic.go:356] "Generic (PLEG): container finished" podID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerID="af684e2c1436cd6d1b9167b9df67c73f4ff13d1291180b93b7e3831822dd8978" exitCode=60 Oct 30 19:00:04.384885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.384511 4862 scope.go:122] "RemoveContainer" containerID="af684e2c1436cd6d1b9167b9df67c73f4ff13d1291180b93b7e3831822dd8978" Oct 30 19:00:04.384885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.384827 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" event={"ID":"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7","Type":"ContainerDied","Data":"af684e2c1436cd6d1b9167b9df67c73f4ff13d1291180b93b7e3831822dd8978"} Oct 30 19:00:04.396490 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.396460 4862 generic.go:356] "Generic (PLEG): container finished" podID="1efeda39-fc25-4232-8bba-e023790536cc" containerID="edea5970407918b20f90a73aa69c9fe7d866cf73972365c673065725bb47b109" exitCode=60 Oct 30 19:00:04.396609 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.396517 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" event={"ID":"1efeda39-fc25-4232-8bba-e023790536cc","Type":"ContainerDied","Data":"edea5970407918b20f90a73aa69c9fe7d866cf73972365c673065725bb47b109"} Oct 30 19:00:04.402916 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:04.397087 4862 scope.go:122] "RemoveContainer" containerID="edea5970407918b20f90a73aa69c9fe7d866cf73972365c673065725bb47b109" Oct 30 19:00:05.413179 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.413146 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" event={"ID":"5edcb24a-8fa2-4038-a6a6-778febfd0a7d","Type":"ContainerStarted","Data":"9539f51e4a1385fc0b7bc4acc909f449a7d6041b14f9e96f3032bd91a0ea102d"} Oct 30 19:00:05.415027 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.414992 4862 generic.go:356] "Generic (PLEG): container finished" podID="7d313143-8642-4426-8660-66d58f0702b3" containerID="173b8afc3d0ef67745d843fe89139b99f11263c526bb028c1fe99c31ab02896f" exitCode=60 Oct 30 19:00:05.415168 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.415068 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" event={"ID":"7d313143-8642-4426-8660-66d58f0702b3","Type":"ContainerDied","Data":"173b8afc3d0ef67745d843fe89139b99f11263c526bb028c1fe99c31ab02896f"} Oct 30 19:00:05.415396 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.415377 4862 scope.go:122] "RemoveContainer" containerID="173b8afc3d0ef67745d843fe89139b99f11263c526bb028c1fe99c31ab02896f" Oct 30 19:00:05.419059 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.419037 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" event={"ID":"cfdf45ce-60c7-44a1-91bf-21236823cbff","Type":"ContainerStarted","Data":"c3080cdac51ecb07f583e6eb47ab97439a23aa947d1c3b9642229b7d388552da"} Oct 30 19:00:05.421216 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.421179 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" event={"ID":"8a8916eb-d97c-414b-ad8c-03dfda5aebc6","Type":"ContainerStarted","Data":"afd3ff892177e9c5ac77ed61844dc3948e8d5441d077b78bb467498d0f352dda"} Oct 30 19:00:05.423692 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.423664 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" event={"ID":"0bb79e7b-310f-4013-9d4e-387a6433ba9f","Type":"ContainerStarted","Data":"d3cb3d0da15aa8818f1f7ed0adea6078e30a28131c8d1062a8a6ffe87fea1677"} Oct 30 19:00:05.426172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.426137 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" event={"ID":"5704fe4a-6a39-418d-93ef-c5080c48b9a5","Type":"ContainerStarted","Data":"9980d02305bbdd988cbb81c7fd09d41529e0b4017d25243da1719a0c691dd835"} Oct 30 19:00:05.428191 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.428169 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" event={"ID":"09356c3c-e94c-4a79-855a-fc550e0eb5f1","Type":"ContainerStarted","Data":"41d625549bed7d823b8c600b33e822d46892b54cb45c577ff74df1bb7f071116"} Oct 30 19:00:05.432237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.432211 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" event={"ID":"d9f5496e-2536-4aae-a65e-76e2bac7b327","Type":"ContainerStarted","Data":"d2a8efd5749b3b24086dbfb0f47c599115d5022031dbcc362b23c68352e0559b"} Oct 30 19:00:05.434412 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.434390 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" event={"ID":"8888996a-7141-4f3f-83b3-d87811232f6a","Type":"ContainerStarted","Data":"ca2e0f3441c0de309d016d0a9569934a0a221bb8af9953f1967880862d22b863"} Oct 30 19:00:05.493867 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.493814 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" podStartSLOduration=5.493801138 podStartE2EDuration="5.493801138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.493018581 +0000 UTC m=+3704.542063494" watchObservedRunningTime="2026-10-30 19:00:05.493801138 +0000 UTC m=+3704.542846051" Oct 30 19:00:05.543928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.539556 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" podStartSLOduration=5.539538854 podStartE2EDuration="5.539538854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.531103139 +0000 UTC m=+3704.580148063" watchObservedRunningTime="2026-10-30 19:00:05.539538854 +0000 UTC m=+3704.588583768" Oct 30 19:00:05.615877 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.615829 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" podStartSLOduration=5.615817269 podStartE2EDuration="5.615817269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.611156477 +0000 UTC m=+3704.660201406" watchObservedRunningTime="2026-10-30 19:00:05.615817269 +0000 UTC m=+3704.664862182" Oct 30 19:00:05.657177 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.657128 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" podStartSLOduration=5.657115341 podStartE2EDuration="5.657115341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.651174479 +0000 UTC m=+3704.700219389" watchObservedRunningTime="2026-10-30 19:00:05.657115341 +0000 UTC m=+3704.706160248" Oct 30 19:00:05.693765 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.692980 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" podStartSLOduration=5.692961855 podStartE2EDuration="5.692961855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.690076792 +0000 UTC m=+3704.739121709" watchObservedRunningTime="2026-10-30 19:00:05.692961855 +0000 UTC m=+3704.742006769" Oct 30 19:00:05.770789 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:05.770735 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" podStartSLOduration=5.7707204690000005 podStartE2EDuration="5.770720469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:05.768282448 +0000 UTC m=+3704.817327355" watchObservedRunningTime="2026-10-30 19:00:05.770720469 +0000 UTC m=+3704.819765383" Oct 30 19:00:05.797121 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:05.797087 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfdf45ce_60c7_44a1_91bf_21236823cbff.slice/crio-c3080cdac51ecb07f583e6eb47ab97439a23aa947d1c3b9642229b7d388552da.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:06.446398 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.446366 4862 generic.go:356] "Generic (PLEG): container finished" podID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerID="41d625549bed7d823b8c600b33e822d46892b54cb45c577ff74df1bb7f071116" exitCode=60 Oct 30 19:00:06.446833 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.446453 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" event={"ID":"09356c3c-e94c-4a79-855a-fc550e0eb5f1","Type":"ContainerDied","Data":"41d625549bed7d823b8c600b33e822d46892b54cb45c577ff74df1bb7f071116"} Oct 30 19:00:06.446833 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.446536 4862 scope.go:122] "RemoveContainer" containerID="b780c69b8828f2a127abe5590295a5fee76e1ba4ea4c343c84e04f7c42e5de8d" Oct 30 19:00:06.446833 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.446717 4862 scope.go:122] "RemoveContainer" containerID="41d625549bed7d823b8c600b33e822d46892b54cb45c577ff74df1bb7f071116" Oct 30 19:00:06.447169 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.447143 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv_odh-ai-gateway-infra(09356c3c-e94c-4a79-855a-fc550e0eb5f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" Oct 30 19:00:06.448786 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.448763 4862 generic.go:356] "Generic (PLEG): container finished" podID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerID="bc2aebe69a29d6822a7dcf814092ca99034848a772e19aa5a3cf95108ed3e085" exitCode=60 Oct 30 19:00:06.448886 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.448841 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" event={"ID":"ad42ac0a-7ac9-4996-a585-a77c79873164","Type":"ContainerDied","Data":"bc2aebe69a29d6822a7dcf814092ca99034848a772e19aa5a3cf95108ed3e085"} Oct 30 19:00:06.449172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.449155 4862 scope.go:122] "RemoveContainer" containerID="bc2aebe69a29d6822a7dcf814092ca99034848a772e19aa5a3cf95108ed3e085" Oct 30 19:00:06.449525 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.449506 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp_odh-ai-gateway-infra(ad42ac0a-7ac9-4996-a585-a77c79873164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" Oct 30 19:00:06.450858 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.450835 4862 generic.go:356] "Generic (PLEG): container finished" podID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerID="3ca943ed72b3107dcceffb73298acceb5e982ec166411982492c56a24656e401" exitCode=60 Oct 30 19:00:06.450955 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.450886 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" event={"ID":"b4ea8704-add1-4406-8aa4-d46f42e68614","Type":"ContainerDied","Data":"3ca943ed72b3107dcceffb73298acceb5e982ec166411982492c56a24656e401"} Oct 30 19:00:06.451173 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.451156 4862 scope.go:122] "RemoveContainer" containerID="3ca943ed72b3107dcceffb73298acceb5e982ec166411982492c56a24656e401" Oct 30 19:00:06.451491 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.451470 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9_odh-ai-gateway-infra(b4ea8704-add1-4406-8aa4-d46f42e68614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" Oct 30 19:00:06.453153 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.453129 4862 generic.go:356] "Generic (PLEG): container finished" podID="38926db9-ad05-4840-86a2-8f066e0de74c" containerID="0cc795381ad261c214ffbd6d2e1fb729623e8b4d8d1d201e24c8efa1157a4a8e" exitCode=60 Oct 30 19:00:06.453290 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.453234 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" event={"ID":"38926db9-ad05-4840-86a2-8f066e0de74c","Type":"ContainerDied","Data":"0cc795381ad261c214ffbd6d2e1fb729623e8b4d8d1d201e24c8efa1157a4a8e"} Oct 30 19:00:06.453457 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.453443 4862 scope.go:122] "RemoveContainer" containerID="0cc795381ad261c214ffbd6d2e1fb729623e8b4d8d1d201e24c8efa1157a4a8e" Oct 30 19:00:06.453728 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.453706 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz_odh-ai-gateway-infra(38926db9-ad05-4840-86a2-8f066e0de74c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" Oct 30 19:00:06.455327 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.455308 4862 generic.go:356] "Generic (PLEG): container finished" podID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerID="e09261643d31463f4a1d60d7aa49ec69243bf269c2b63112716763155b176bdc" exitCode=60 Oct 30 19:00:06.455426 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.455381 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" event={"ID":"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8","Type":"ContainerDied","Data":"e09261643d31463f4a1d60d7aa49ec69243bf269c2b63112716763155b176bdc"} Oct 30 19:00:06.455605 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.455592 4862 scope.go:122] "RemoveContainer" containerID="e09261643d31463f4a1d60d7aa49ec69243bf269c2b63112716763155b176bdc" Oct 30 19:00:06.455881 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.455859 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7_odh-ai-gateway-infra(e365121d-8f28-44c0-8b9c-3ce63f3b8fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" Oct 30 19:00:06.457313 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.457290 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" event={"ID":"72239397-4285-47ff-b37f-c42943e13b82","Type":"ContainerStarted","Data":"51f5de507bee191f9253f5277ea597c672f9cdd17233b9e789181d1171b2b5a2"} Oct 30 19:00:06.459040 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.459021 4862 generic.go:356] "Generic (PLEG): container finished" podID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerID="d2a8efd5749b3b24086dbfb0f47c599115d5022031dbcc362b23c68352e0559b" exitCode=60 Oct 30 19:00:06.459136 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.459087 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" event={"ID":"d9f5496e-2536-4aae-a65e-76e2bac7b327","Type":"ContainerDied","Data":"d2a8efd5749b3b24086dbfb0f47c599115d5022031dbcc362b23c68352e0559b"} Oct 30 19:00:06.459308 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.459294 4862 scope.go:122] "RemoveContainer" containerID="d2a8efd5749b3b24086dbfb0f47c599115d5022031dbcc362b23c68352e0559b" Oct 30 19:00:06.459542 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.459518 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr_odh-ai-gateway-infra(d9f5496e-2536-4aae-a65e-76e2bac7b327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" Oct 30 19:00:06.460976 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.460958 4862 generic.go:356] "Generic (PLEG): container finished" podID="8888996a-7141-4f3f-83b3-d87811232f6a" containerID="ca2e0f3441c0de309d016d0a9569934a0a221bb8af9953f1967880862d22b863" exitCode=60 Oct 30 19:00:06.461081 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.461045 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" event={"ID":"8888996a-7141-4f3f-83b3-d87811232f6a","Type":"ContainerDied","Data":"ca2e0f3441c0de309d016d0a9569934a0a221bb8af9953f1967880862d22b863"} Oct 30 19:00:06.461210 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.461196 4862 scope.go:122] "RemoveContainer" containerID="ca2e0f3441c0de309d016d0a9569934a0a221bb8af9953f1967880862d22b863" Oct 30 19:00:06.461482 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.461464 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2_odh-ai-gateway-infra(8888996a-7141-4f3f-83b3-d87811232f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" Oct 30 19:00:06.462939 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.462920 4862 generic.go:356] "Generic (PLEG): container finished" podID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerID="9539f51e4a1385fc0b7bc4acc909f449a7d6041b14f9e96f3032bd91a0ea102d" exitCode=60 Oct 30 19:00:06.463033 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.462994 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" event={"ID":"5edcb24a-8fa2-4038-a6a6-778febfd0a7d","Type":"ContainerDied","Data":"9539f51e4a1385fc0b7bc4acc909f449a7d6041b14f9e96f3032bd91a0ea102d"} Oct 30 19:00:06.463198 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.463182 4862 scope.go:122] "RemoveContainer" containerID="9539f51e4a1385fc0b7bc4acc909f449a7d6041b14f9e96f3032bd91a0ea102d" Oct 30 19:00:06.463523 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.463490 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz_odh-ai-gateway-infra(5edcb24a-8fa2-4038-a6a6-778febfd0a7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" Oct 30 19:00:06.464820 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.464802 4862 generic.go:356] "Generic (PLEG): container finished" podID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerID="c3080cdac51ecb07f583e6eb47ab97439a23aa947d1c3b9642229b7d388552da" exitCode=60 Oct 30 19:00:06.464921 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.464883 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" event={"ID":"cfdf45ce-60c7-44a1-91bf-21236823cbff","Type":"ContainerDied","Data":"c3080cdac51ecb07f583e6eb47ab97439a23aa947d1c3b9642229b7d388552da"} Oct 30 19:00:06.465077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.465061 4862 scope.go:122] "RemoveContainer" containerID="c3080cdac51ecb07f583e6eb47ab97439a23aa947d1c3b9642229b7d388552da" Oct 30 19:00:06.465334 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.465313 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w_odh-ai-gateway-infra(cfdf45ce-60c7-44a1-91bf-21236823cbff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" Oct 30 19:00:06.466830 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.466810 4862 generic.go:356] "Generic (PLEG): container finished" podID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerID="11037ee773c3744b4db3200f3f24d45d7670d00a6fd59cf66870d2612a4e31d3" exitCode=60 Oct 30 19:00:06.466914 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.466833 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" event={"ID":"1d6fae04-3d28-4cf2-a8d5-a58c9be23015","Type":"ContainerDied","Data":"11037ee773c3744b4db3200f3f24d45d7670d00a6fd59cf66870d2612a4e31d3"} Oct 30 19:00:06.467115 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.467099 4862 scope.go:122] "RemoveContainer" containerID="11037ee773c3744b4db3200f3f24d45d7670d00a6fd59cf66870d2612a4e31d3" Oct 30 19:00:06.467400 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.467375 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z_odh-ai-gateway-infra(1d6fae04-3d28-4cf2-a8d5-a58c9be23015)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" Oct 30 19:00:06.468789 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.468769 4862 generic.go:356] "Generic (PLEG): container finished" podID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerID="afd3ff892177e9c5ac77ed61844dc3948e8d5441d077b78bb467498d0f352dda" exitCode=60 Oct 30 19:00:06.468877 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.468832 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" event={"ID":"8a8916eb-d97c-414b-ad8c-03dfda5aebc6","Type":"ContainerDied","Data":"afd3ff892177e9c5ac77ed61844dc3948e8d5441d077b78bb467498d0f352dda"} Oct 30 19:00:06.469010 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.468996 4862 scope.go:122] "RemoveContainer" containerID="afd3ff892177e9c5ac77ed61844dc3948e8d5441d077b78bb467498d0f352dda" Oct 30 19:00:06.469240 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.469223 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d_odh-ai-gateway-infra(8a8916eb-d97c-414b-ad8c-03dfda5aebc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" Oct 30 19:00:06.470882 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.470863 4862 generic.go:356] "Generic (PLEG): container finished" podID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerID="d3cb3d0da15aa8818f1f7ed0adea6078e30a28131c8d1062a8a6ffe87fea1677" exitCode=60 Oct 30 19:00:06.470980 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.470921 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" event={"ID":"0bb79e7b-310f-4013-9d4e-387a6433ba9f","Type":"ContainerDied","Data":"d3cb3d0da15aa8818f1f7ed0adea6078e30a28131c8d1062a8a6ffe87fea1677"} Oct 30 19:00:06.471178 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.471164 4862 scope.go:122] "RemoveContainer" containerID="d3cb3d0da15aa8818f1f7ed0adea6078e30a28131c8d1062a8a6ffe87fea1677" Oct 30 19:00:06.471417 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.471396 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4_odh-ai-gateway-infra(0bb79e7b-310f-4013-9d4e-387a6433ba9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" Oct 30 19:00:06.472972 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.472951 4862 generic.go:356] "Generic (PLEG): container finished" podID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerID="97df060a26f2f90c71f7c50123a6bc4ccb60ca46c05e8d9cc9567b7f981c772e" exitCode=60 Oct 30 19:00:06.473066 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.472975 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" event={"ID":"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7","Type":"ContainerDied","Data":"97df060a26f2f90c71f7c50123a6bc4ccb60ca46c05e8d9cc9567b7f981c772e"} Oct 30 19:00:06.473351 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.473331 4862 scope.go:122] "RemoveContainer" containerID="97df060a26f2f90c71f7c50123a6bc4ccb60ca46c05e8d9cc9567b7f981c772e" Oct 30 19:00:06.473624 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.473599 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv_odh-ai-gateway-infra(3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" Oct 30 19:00:06.475293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.475257 4862 generic.go:356] "Generic (PLEG): container finished" podID="1efeda39-fc25-4232-8bba-e023790536cc" containerID="de816ecf95c63594bf406941a18ae74a13f9ce9c5341a71d611bf8cbe83b7bb6" exitCode=60 Oct 30 19:00:06.475407 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.475348 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" event={"ID":"1efeda39-fc25-4232-8bba-e023790536cc","Type":"ContainerDied","Data":"de816ecf95c63594bf406941a18ae74a13f9ce9c5341a71d611bf8cbe83b7bb6"} Oct 30 19:00:06.475604 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.475583 4862 scope.go:122] "RemoveContainer" containerID="de816ecf95c63594bf406941a18ae74a13f9ce9c5341a71d611bf8cbe83b7bb6" Oct 30 19:00:06.475864 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.475845 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7_odh-ai-gateway-infra(1efeda39-fc25-4232-8bba-e023790536cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" podUID="1efeda39-fc25-4232-8bba-e023790536cc" Oct 30 19:00:06.477516 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.477494 4862 generic.go:356] "Generic (PLEG): container finished" podID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerID="9980d02305bbdd988cbb81c7fd09d41529e0b4017d25243da1719a0c691dd835" exitCode=60 Oct 30 19:00:06.477606 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.477542 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" event={"ID":"5704fe4a-6a39-418d-93ef-c5080c48b9a5","Type":"ContainerDied","Data":"9980d02305bbdd988cbb81c7fd09d41529e0b4017d25243da1719a0c691dd835"} Oct 30 19:00:06.477793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.477774 4862 scope.go:122] "RemoveContainer" containerID="9980d02305bbdd988cbb81c7fd09d41529e0b4017d25243da1719a0c691dd835" Oct 30 19:00:06.478151 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:06.478124 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj_odh-ai-gateway-infra(5704fe4a-6a39-418d-93ef-c5080c48b9a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" Oct 30 19:00:06.482435 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.482417 4862 scope.go:122] "RemoveContainer" containerID="3848278eafc07d64524606439af116f6dc08b98001d5798c85e1d0e6857afeb7" Oct 30 19:00:06.498151 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.498134 4862 scope.go:122] "RemoveContainer" containerID="1e26877cb29979861360cc23f8984db8deab73e97497243fd79986762799c028" Oct 30 19:00:06.512764 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.512727 4862 scope.go:122] "RemoveContainer" containerID="dae406a0a961a97910c126d142274a7c41a6eef4650b290e79a87126b58974d8" Oct 30 19:00:06.531394 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.530520 4862 scope.go:122] "RemoveContainer" containerID="784496bd98f483b174455d31d5102598e92618f42f6bc83ad38f4fc4a0638cea" Oct 30 19:00:06.549522 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.549505 4862 scope.go:122] "RemoveContainer" containerID="c443979759772e70b3eda2de2e4a1a23f59aa5c687c495cce273aa9bdb9f0873" Oct 30 19:00:06.566921 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.566893 4862 scope.go:122] "RemoveContainer" containerID="ad17b2f96d89fc34939452b758b6d3bbce3639b6749ef86cf9b193105db0646d" Oct 30 19:00:06.587569 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.587549 4862 scope.go:122] "RemoveContainer" containerID="5a0746e5900e0e3935d8d836ae01d210623d14b29f0c2ce0e670a5a82711df48" Oct 30 19:00:06.602693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.602674 4862 scope.go:122] "RemoveContainer" containerID="0a12037dbb8b0ee25915471900d8cbafbd9ced0c825e9ab688de205c5b2bc88c" Oct 30 19:00:06.617439 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.617417 4862 scope.go:122] "RemoveContainer" containerID="01f002a1963de1c842318736a789293bc5aea59b6720720ab1d38a5d742650ea" Oct 30 19:00:06.633927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.633898 4862 scope.go:122] "RemoveContainer" containerID="7b1dcd73017e8f48be905f9729fff821f203d895c6c202ddec0069b72b6e2814" Oct 30 19:00:06.648813 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.648794 4862 scope.go:122] "RemoveContainer" containerID="b1491fb20fd6b111a9c5c91a2d57f382544d7399c639b316031d0491c6bc4b76" Oct 30 19:00:06.663728 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.663710 4862 scope.go:122] "RemoveContainer" containerID="af684e2c1436cd6d1b9167b9df67c73f4ff13d1291180b93b7e3831822dd8978" Oct 30 19:00:06.677375 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.677341 4862 scope.go:122] "RemoveContainer" containerID="edea5970407918b20f90a73aa69c9fe7d866cf73972365c673065725bb47b109" Oct 30 19:00:06.691785 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:06.691754 4862 scope.go:122] "RemoveContainer" containerID="3afa152a0d2d16560941a441067296356371c084e8896fc386661d8dbdf659d2" Oct 30 19:00:07.488951 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.488924 4862 scope.go:122] "RemoveContainer" containerID="41d625549bed7d823b8c600b33e822d46892b54cb45c577ff74df1bb7f071116" Oct 30 19:00:07.489516 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.489399 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv_odh-ai-gateway-infra(09356c3c-e94c-4a79-855a-fc550e0eb5f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" Oct 30 19:00:07.491587 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.491565 4862 scope.go:122] "RemoveContainer" containerID="bc2aebe69a29d6822a7dcf814092ca99034848a772e19aa5a3cf95108ed3e085" Oct 30 19:00:07.491946 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.491911 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp_odh-ai-gateway-infra(ad42ac0a-7ac9-4996-a585-a77c79873164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" Oct 30 19:00:07.493675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.493645 4862 scope.go:122] "RemoveContainer" containerID="3ca943ed72b3107dcceffb73298acceb5e982ec166411982492c56a24656e401" Oct 30 19:00:07.493957 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.493936 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9_odh-ai-gateway-infra(b4ea8704-add1-4406-8aa4-d46f42e68614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" Oct 30 19:00:07.496586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.496566 4862 scope.go:122] "RemoveContainer" containerID="0cc795381ad261c214ffbd6d2e1fb729623e8b4d8d1d201e24c8efa1157a4a8e" Oct 30 19:00:07.496844 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.496822 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz_odh-ai-gateway-infra(38926db9-ad05-4840-86a2-8f066e0de74c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" Oct 30 19:00:07.498900 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.498878 4862 scope.go:122] "RemoveContainer" containerID="e09261643d31463f4a1d60d7aa49ec69243bf269c2b63112716763155b176bdc" Oct 30 19:00:07.499177 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.499155 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7_odh-ai-gateway-infra(e365121d-8f28-44c0-8b9c-3ce63f3b8fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" Oct 30 19:00:07.500857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.500832 4862 generic.go:356] "Generic (PLEG): container finished" podID="72239397-4285-47ff-b37f-c42943e13b82" containerID="51f5de507bee191f9253f5277ea597c672f9cdd17233b9e789181d1171b2b5a2" exitCode=60 Oct 30 19:00:07.500966 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.500949 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" event={"ID":"72239397-4285-47ff-b37f-c42943e13b82","Type":"ContainerDied","Data":"51f5de507bee191f9253f5277ea597c672f9cdd17233b9e789181d1171b2b5a2"} Oct 30 19:00:07.501039 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.500994 4862 scope.go:122] "RemoveContainer" containerID="c7c8817c602a0b394df7ec74c018a3acf88d5109f00dd1503d4b3470a39d5915" Oct 30 19:00:07.501188 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.501172 4862 scope.go:122] "RemoveContainer" containerID="51f5de507bee191f9253f5277ea597c672f9cdd17233b9e789181d1171b2b5a2" Oct 30 19:00:07.501492 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.501460 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8_odh-ai-gateway-infra(72239397-4285-47ff-b37f-c42943e13b82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" podUID="72239397-4285-47ff-b37f-c42943e13b82" Oct 30 19:00:07.503611 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.503589 4862 scope.go:122] "RemoveContainer" containerID="d2a8efd5749b3b24086dbfb0f47c599115d5022031dbcc362b23c68352e0559b" Oct 30 19:00:07.503852 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.503829 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr_odh-ai-gateway-infra(d9f5496e-2536-4aae-a65e-76e2bac7b327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" Oct 30 19:00:07.505825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.505798 4862 scope.go:122] "RemoveContainer" containerID="ca2e0f3441c0de309d016d0a9569934a0a221bb8af9953f1967880862d22b863" Oct 30 19:00:07.506082 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.506059 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2_odh-ai-gateway-infra(8888996a-7141-4f3f-83b3-d87811232f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" Oct 30 19:00:07.508657 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.508636 4862 scope.go:122] "RemoveContainer" containerID="9539f51e4a1385fc0b7bc4acc909f449a7d6041b14f9e96f3032bd91a0ea102d" Oct 30 19:00:07.509038 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.508885 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz_odh-ai-gateway-infra(5edcb24a-8fa2-4038-a6a6-778febfd0a7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" Oct 30 19:00:07.510847 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.510825 4862 generic.go:356] "Generic (PLEG): container finished" podID="7d313143-8642-4426-8660-66d58f0702b3" containerID="9ad0c83efcb88f8ebe003294409ddf7bacd810ec2f0f3a35ad612985f8131741" exitCode=60 Oct 30 19:00:07.510970 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.510894 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" event={"ID":"7d313143-8642-4426-8660-66d58f0702b3","Type":"ContainerDied","Data":"9ad0c83efcb88f8ebe003294409ddf7bacd810ec2f0f3a35ad612985f8131741"} Oct 30 19:00:07.511102 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.511086 4862 scope.go:122] "RemoveContainer" containerID="9ad0c83efcb88f8ebe003294409ddf7bacd810ec2f0f3a35ad612985f8131741" Oct 30 19:00:07.511362 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.511343 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk_odh-ai-gateway-infra(7d313143-8642-4426-8660-66d58f0702b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" podUID="7d313143-8642-4426-8660-66d58f0702b3" Oct 30 19:00:07.513006 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.512989 4862 scope.go:122] "RemoveContainer" containerID="c3080cdac51ecb07f583e6eb47ab97439a23aa947d1c3b9642229b7d388552da" Oct 30 19:00:07.513253 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.513207 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w_odh-ai-gateway-infra(cfdf45ce-60c7-44a1-91bf-21236823cbff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" Oct 30 19:00:07.519773 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.519749 4862 scope.go:122] "RemoveContainer" containerID="11037ee773c3744b4db3200f3f24d45d7670d00a6fd59cf66870d2612a4e31d3" Oct 30 19:00:07.520046 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.520021 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z_odh-ai-gateway-infra(1d6fae04-3d28-4cf2-a8d5-a58c9be23015)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" Oct 30 19:00:07.524123 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.523446 4862 scope.go:122] "RemoveContainer" containerID="173b8afc3d0ef67745d843fe89139b99f11263c526bb028c1fe99c31ab02896f" Oct 30 19:00:07.566243 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.566215 4862 scope.go:122] "RemoveContainer" containerID="afd3ff892177e9c5ac77ed61844dc3948e8d5441d077b78bb467498d0f352dda" Oct 30 19:00:07.566564 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.566535 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d_odh-ai-gateway-infra(8a8916eb-d97c-414b-ad8c-03dfda5aebc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" Oct 30 19:00:07.576246 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.576221 4862 scope.go:122] "RemoveContainer" containerID="d3cb3d0da15aa8818f1f7ed0adea6078e30a28131c8d1062a8a6ffe87fea1677" Oct 30 19:00:07.576575 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.576551 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4_odh-ai-gateway-infra(0bb79e7b-310f-4013-9d4e-387a6433ba9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" Oct 30 19:00:07.581192 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.581136 4862 scope.go:122] "RemoveContainer" containerID="97df060a26f2f90c71f7c50123a6bc4ccb60ca46c05e8d9cc9567b7f981c772e" Oct 30 19:00:07.581450 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.581428 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv_odh-ai-gateway-infra(3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" Oct 30 19:00:07.583391 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.583374 4862 scope.go:122] "RemoveContainer" containerID="de816ecf95c63594bf406941a18ae74a13f9ce9c5341a71d611bf8cbe83b7bb6" Oct 30 19:00:07.583651 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.583627 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7_odh-ai-gateway-infra(1efeda39-fc25-4232-8bba-e023790536cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" podUID="1efeda39-fc25-4232-8bba-e023790536cc" Oct 30 19:00:07.585111 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:07.585088 4862 scope.go:122] "RemoveContainer" containerID="9980d02305bbdd988cbb81c7fd09d41529e0b4017d25243da1719a0c691dd835" Oct 30 19:00:07.585372 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:07.585350 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj_odh-ai-gateway-infra(5704fe4a-6a39-418d-93ef-c5080c48b9a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" Oct 30 19:00:08.596738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:08.596708 4862 scope.go:122] "RemoveContainer" containerID="51f5de507bee191f9253f5277ea597c672f9cdd17233b9e789181d1171b2b5a2" Oct 30 19:00:08.597236 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:08.597148 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8_odh-ai-gateway-infra(72239397-4285-47ff-b37f-c42943e13b82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" podUID="72239397-4285-47ff-b37f-c42943e13b82" Oct 30 19:00:08.598728 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:08.598707 4862 scope.go:122] "RemoveContainer" containerID="9ad0c83efcb88f8ebe003294409ddf7bacd810ec2f0f3a35ad612985f8131741" Oct 30 19:00:08.598968 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:08.598949 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk_odh-ai-gateway-infra(7d313143-8642-4426-8660-66d58f0702b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" podUID="7d313143-8642-4426-8660-66d58f0702b3" Oct 30 19:00:08.690942 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:08.690908 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" podUID="abd590d7-b8b0-4356-9566-c3e62499c87b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:00:08.690942 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:08.690936 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" podUID="6e82685b-9c65-432b-a863-c39edd6950e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:00:12.292464 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.292433 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 19:00:12.368722 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.364138 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e82685b-9c65-432b-a863-c39edd6950e4-kube-api-access-jqddf\" (UniqueName: \"kubernetes.io/projected/6e82685b-9c65-432b-a863-c39edd6950e4-kube-api-access-jqddf\") pod \"6e82685b-9c65-432b-a863-c39edd6950e4\" (UID: \"6e82685b-9c65-432b-a863-c39edd6950e4\") " Oct 30 19:00:12.368722 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.364423 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e82685b-9c65-432b-a863-c39edd6950e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e82685b-9c65-432b-a863-c39edd6950e4-plugins-config-volume\") pod \"6e82685b-9c65-432b-a863-c39edd6950e4\" (UID: \"6e82685b-9c65-432b-a863-c39edd6950e4\") " Oct 30 19:00:12.370806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.369155 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e82685b-9c65-432b-a863-c39edd6950e4-plugins-config-volume" pod "6e82685b-9c65-432b-a863-c39edd6950e4" (UID: "6e82685b-9c65-432b-a863-c39edd6950e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:12.372710 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.372551 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 19:00:12.373726 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.373700 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e82685b-9c65-432b-a863-c39edd6950e4-kube-api-access-jqddf" pod "6e82685b-9c65-432b-a863-c39edd6950e4" (UID: "6e82685b-9c65-432b-a863-c39edd6950e4"). InnerVolumeSpecName "kube-api-access-jqddf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:12.498337 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.471040 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd590d7-b8b0-4356-9566-c3e62499c87b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd590d7-b8b0-4356-9566-c3e62499c87b-plugins-config-volume\") pod \"abd590d7-b8b0-4356-9566-c3e62499c87b\" (UID: \"abd590d7-b8b0-4356-9566-c3e62499c87b\") " Oct 30 19:00:12.498337 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.471144 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd590d7-b8b0-4356-9566-c3e62499c87b-kube-api-access-kbpzj\" (UniqueName: \"kubernetes.io/projected/abd590d7-b8b0-4356-9566-c3e62499c87b-kube-api-access-kbpzj\") pod \"abd590d7-b8b0-4356-9566-c3e62499c87b\" (UID: \"abd590d7-b8b0-4356-9566-c3e62499c87b\") " Oct 30 19:00:12.498337 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.472183 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e82685b-9c65-432b-a863-c39edd6950e4-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:12.498337 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.472209 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqddf\" (UniqueName: \"kubernetes.io/projected/6e82685b-9c65-432b-a863-c39edd6950e4-kube-api-access-jqddf\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:12.512170 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.499178 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd590d7-b8b0-4356-9566-c3e62499c87b-plugins-config-volume" pod "abd590d7-b8b0-4356-9566-c3e62499c87b" (UID: "abd590d7-b8b0-4356-9566-c3e62499c87b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:12.512729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.512697 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd590d7-b8b0-4356-9566-c3e62499c87b-kube-api-access-kbpzj" pod "abd590d7-b8b0-4356-9566-c3e62499c87b" (UID: "abd590d7-b8b0-4356-9566-c3e62499c87b"). InnerVolumeSpecName "kube-api-access-kbpzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:12.579436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.579371 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd590d7-b8b0-4356-9566-c3e62499c87b-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:12.579436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.579407 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpzj\" (UniqueName: \"kubernetes.io/projected/abd590d7-b8b0-4356-9566-c3e62499c87b-kube-api-access-kbpzj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:12.638681 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.638651 4862 generic.go:356] "Generic (PLEG): container finished" podID="abd590d7-b8b0-4356-9566-c3e62499c87b" containerID="16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11" exitCode=0 Oct 30 19:00:12.638880 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.638703 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" Oct 30 19:00:12.638880 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.638738 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" event={"ID":"abd590d7-b8b0-4356-9566-c3e62499c87b","Type":"ContainerDied","Data":"16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11"} Oct 30 19:00:12.638880 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.638804 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8" event={"ID":"abd590d7-b8b0-4356-9566-c3e62499c87b","Type":"ContainerDied","Data":"73953f4c08394be8186de24f3c6f6f7aab4f8fe27d741e983a1bc5fcc00f89e3"} Oct 30 19:00:12.638880 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.638837 4862 scope.go:122] "RemoveContainer" containerID="16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11" Oct 30 19:00:12.640424 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.640406 4862 generic.go:356] "Generic (PLEG): container finished" podID="6e82685b-9c65-432b-a863-c39edd6950e4" containerID="a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1" exitCode=0 Oct 30 19:00:12.640533 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.640459 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" Oct 30 19:00:12.640533 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.640468 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" event={"ID":"6e82685b-9c65-432b-a863-c39edd6950e4","Type":"ContainerDied","Data":"a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1"} Oct 30 19:00:12.640533 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.640523 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7" event={"ID":"6e82685b-9c65-432b-a863-c39edd6950e4","Type":"ContainerDied","Data":"1e6d8d4aec4919af41a6d1d2fed0d01d41da53925c08bc88b9af62227dfd55ad"} Oct 30 19:00:12.654322 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.654304 4862 scope.go:122] "RemoveContainer" containerID="16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11" Oct 30 19:00:12.654613 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:12.654593 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11\": container with ID starting with 16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11 not found: ID does not exist" containerID="16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11" Oct 30 19:00:12.654669 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.654620 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11"} err="failed to get container status \"16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11\": rpc error: code = NotFound desc = could not find container \"16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11\": container with ID starting with 16342d91e727b9303d413b01c4989b189b562e32fd3a9a04352fbf12fa0acc11 not found: ID does not exist" Oct 30 19:00:12.654669 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.654637 4862 scope.go:122] "RemoveContainer" containerID="a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1" Oct 30 19:00:12.667079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.667057 4862 scope.go:122] "RemoveContainer" containerID="a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1" Oct 30 19:00:12.667443 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:12.667416 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1\": container with ID starting with a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1 not found: ID does not exist" containerID="a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1" Oct 30 19:00:12.667524 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.667455 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1"} err="failed to get container status \"a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1\": rpc error: code = NotFound desc = could not find container \"a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1\": container with ID starting with a0f7954aaea54caa4476e9c7a99e2d99756377fb176f3c4b10475cf7b10d6ef1 not found: ID does not exist" Oct 30 19:00:12.697563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.697536 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8"] Oct 30 19:00:12.710408 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.710384 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c52ec7-6977fb4447-zvsc8"] Oct 30 19:00:12.742988 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.742962 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7"] Oct 30 19:00:12.747520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:12.747500 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c52ec7-f6884465b-465q7"] Oct 30 19:00:13.068796 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:13.068752 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e82685b-9c65-432b-a863-c39edd6950e4" path="/var/lib/kubelet/pods/6e82685b-9c65-432b-a863-c39edd6950e4/volumes" Oct 30 19:00:13.069452 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:13.069429 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd590d7-b8b0-4356-9566-c3e62499c87b" path="/var/lib/kubelet/pods/abd590d7-b8b0-4356-9566-c3e62499c87b/volumes" Oct 30 19:00:16.055130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:16.055101 4862 scope.go:122] "RemoveContainer" containerID="afd3ff892177e9c5ac77ed61844dc3948e8d5441d077b78bb467498d0f352dda" Oct 30 19:00:16.055513 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:16.055163 4862 scope.go:122] "RemoveContainer" containerID="d3cb3d0da15aa8818f1f7ed0adea6078e30a28131c8d1062a8a6ffe87fea1677" Oct 30 19:00:16.681076 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:16.680994 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" event={"ID":"8a8916eb-d97c-414b-ad8c-03dfda5aebc6","Type":"ContainerStarted","Data":"b56c27b27bbb13eb72105793da3ac76c0d98f0a825a67ac75ad790462b903162"} Oct 30 19:00:16.682946 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:16.682920 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" event={"ID":"0bb79e7b-310f-4013-9d4e-387a6433ba9f","Type":"ContainerStarted","Data":"25d589bdc3c373fef51ef9bac0df41065527a2ecfa31bc487c15592e8e2cdd8a"} Oct 30 19:00:17.055286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.055241 4862 scope.go:122] "RemoveContainer" containerID="0cc795381ad261c214ffbd6d2e1fb729623e8b4d8d1d201e24c8efa1157a4a8e" Oct 30 19:00:17.693158 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.693080 4862 generic.go:356] "Generic (PLEG): container finished" podID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerID="b56c27b27bbb13eb72105793da3ac76c0d98f0a825a67ac75ad790462b903162" exitCode=60 Oct 30 19:00:17.693343 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.693158 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" event={"ID":"8a8916eb-d97c-414b-ad8c-03dfda5aebc6","Type":"ContainerDied","Data":"b56c27b27bbb13eb72105793da3ac76c0d98f0a825a67ac75ad790462b903162"} Oct 30 19:00:17.693343 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.693227 4862 scope.go:122] "RemoveContainer" containerID="afd3ff892177e9c5ac77ed61844dc3948e8d5441d077b78bb467498d0f352dda" Oct 30 19:00:17.693629 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.693612 4862 scope.go:122] "RemoveContainer" containerID="b56c27b27bbb13eb72105793da3ac76c0d98f0a825a67ac75ad790462b903162" Oct 30 19:00:17.693931 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:17.693907 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d_odh-ai-gateway-infra(8a8916eb-d97c-414b-ad8c-03dfda5aebc6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" Oct 30 19:00:17.695402 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.695379 4862 generic.go:356] "Generic (PLEG): container finished" podID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerID="25d589bdc3c373fef51ef9bac0df41065527a2ecfa31bc487c15592e8e2cdd8a" exitCode=60 Oct 30 19:00:17.695489 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.695451 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" event={"ID":"0bb79e7b-310f-4013-9d4e-387a6433ba9f","Type":"ContainerDied","Data":"25d589bdc3c373fef51ef9bac0df41065527a2ecfa31bc487c15592e8e2cdd8a"} Oct 30 19:00:17.695771 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.695752 4862 scope.go:122] "RemoveContainer" containerID="25d589bdc3c373fef51ef9bac0df41065527a2ecfa31bc487c15592e8e2cdd8a" Oct 30 19:00:17.696064 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:17.696044 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4_odh-ai-gateway-infra(0bb79e7b-310f-4013-9d4e-387a6433ba9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" Oct 30 19:00:17.697482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.697464 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" event={"ID":"38926db9-ad05-4840-86a2-8f066e0de74c","Type":"ContainerStarted","Data":"bc5c96337dc2919413925608c4880084670b629a5bf9656da2aa2f476b06f9c3"} Oct 30 19:00:17.707805 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.707785 4862 scope.go:122] "RemoveContainer" containerID="d3cb3d0da15aa8818f1f7ed0adea6078e30a28131c8d1062a8a6ffe87fea1677" Oct 30 19:00:17.738241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.738220 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d"] Oct 30 19:00:17.774729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:17.774703 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4"] Oct 30 19:00:18.055083 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.055051 4862 scope.go:122] "RemoveContainer" containerID="d2a8efd5749b3b24086dbfb0f47c599115d5022031dbcc362b23c68352e0559b" Oct 30 19:00:18.055228 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.055186 4862 scope.go:122] "RemoveContainer" containerID="9ad0c83efcb88f8ebe003294409ddf7bacd810ec2f0f3a35ad612985f8131741" Oct 30 19:00:18.055385 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.055362 4862 scope.go:122] "RemoveContainer" containerID="c3080cdac51ecb07f583e6eb47ab97439a23aa947d1c3b9642229b7d388552da" Oct 30 19:00:18.708190 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.708158 4862 generic.go:356] "Generic (PLEG): container finished" podID="38926db9-ad05-4840-86a2-8f066e0de74c" containerID="bc5c96337dc2919413925608c4880084670b629a5bf9656da2aa2f476b06f9c3" exitCode=60 Oct 30 19:00:18.708395 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.708374 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" event={"ID":"38926db9-ad05-4840-86a2-8f066e0de74c","Type":"ContainerDied","Data":"bc5c96337dc2919413925608c4880084670b629a5bf9656da2aa2f476b06f9c3"} Oct 30 19:00:18.708481 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.708439 4862 scope.go:122] "RemoveContainer" containerID="0cc795381ad261c214ffbd6d2e1fb729623e8b4d8d1d201e24c8efa1157a4a8e" Oct 30 19:00:18.708948 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.708927 4862 scope.go:122] "RemoveContainer" containerID="bc5c96337dc2919413925608c4880084670b629a5bf9656da2aa2f476b06f9c3" Oct 30 19:00:18.709377 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:18.709336 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz_odh-ai-gateway-infra(38926db9-ad05-4840-86a2-8f066e0de74c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" Oct 30 19:00:18.712171 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.712102 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" event={"ID":"d9f5496e-2536-4aae-a65e-76e2bac7b327","Type":"ContainerStarted","Data":"690fcdde1284924e2498a484d6a0cfe3a5ebdc1644ff3b4b80b05894956537ce"} Oct 30 19:00:18.714463 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.714439 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" event={"ID":"7d313143-8642-4426-8660-66d58f0702b3","Type":"ContainerStarted","Data":"75bfa8f8ae4ce60adc9d01bd2d271b7704f58eec127ab7c22cccfa19bf6f983e"} Oct 30 19:00:18.716770 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.716739 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" event={"ID":"cfdf45ce-60c7-44a1-91bf-21236823cbff","Type":"ContainerStarted","Data":"190016b014c0ad9aa13a38c4d4c30ed8bb5cdf0abe30fb84d9019f56acc0a47f"} Oct 30 19:00:18.809287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.809242 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz"] Oct 30 19:00:18.839295 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:18.838947 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" podStartSLOduration=18.838926287 podStartE2EDuration="18.838926287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:18.836116442 +0000 UTC m=+3717.885161357" watchObservedRunningTime="2026-10-30 19:00:18.838926287 +0000 UTC m=+3717.887971201" Oct 30 19:00:19.003742 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.003715 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:19.006593 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.006576 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:19.060444 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.060359 4862 scope.go:122] "RemoveContainer" containerID="97df060a26f2f90c71f7c50123a6bc4ccb60ca46c05e8d9cc9567b7f981c772e" Oct 30 19:00:19.062598 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.062246 4862 scope.go:122] "RemoveContainer" containerID="9980d02305bbdd988cbb81c7fd09d41529e0b4017d25243da1719a0c691dd835" Oct 30 19:00:19.062598 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.062511 4862 scope.go:122] "RemoveContainer" containerID="ca2e0f3441c0de309d016d0a9569934a0a221bb8af9953f1967880862d22b863" Oct 30 19:00:19.065903 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.065881 4862 scope.go:122] "RemoveContainer" containerID="9539f51e4a1385fc0b7bc4acc909f449a7d6041b14f9e96f3032bd91a0ea102d" Oct 30 19:00:19.066036 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.066015 4862 scope.go:122] "RemoveContainer" containerID="11037ee773c3744b4db3200f3f24d45d7670d00a6fd59cf66870d2612a4e31d3" Oct 30 19:00:19.066695 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.066351 4862 scope.go:122] "RemoveContainer" containerID="de816ecf95c63594bf406941a18ae74a13f9ce9c5341a71d611bf8cbe83b7bb6" Oct 30 19:00:19.113496 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.113463 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb79e7b-310f-4013-9d4e-387a6433ba9f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb79e7b-310f-4013-9d4e-387a6433ba9f-maas-api-service-ca\") pod \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\" (UID: \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\") " Oct 30 19:00:19.114036 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.114005 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-kube-api-access-gvm46\" (UniqueName: \"kubernetes.io/projected/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-kube-api-access-gvm46\") pod \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\" (UID: \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\") " Oct 30 19:00:19.114235 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.114218 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb79e7b-310f-4013-9d4e-387a6433ba9f-kube-api-access-hvd8r\" (UniqueName: \"kubernetes.io/projected/0bb79e7b-310f-4013-9d4e-387a6433ba9f-kube-api-access-hvd8r\") pod \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\" (UID: \"0bb79e7b-310f-4013-9d4e-387a6433ba9f\") " Oct 30 19:00:19.114530 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.114508 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-maas-api-service-ca\") pod \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\" (UID: \"8a8916eb-d97c-414b-ad8c-03dfda5aebc6\") " Oct 30 19:00:19.115842 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.115745 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-maas-api-service-ca" pod "8a8916eb-d97c-414b-ad8c-03dfda5aebc6" (UID: "8a8916eb-d97c-414b-ad8c-03dfda5aebc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.115981 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.115843 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb79e7b-310f-4013-9d4e-387a6433ba9f-maas-api-service-ca" pod "0bb79e7b-310f-4013-9d4e-387a6433ba9f" (UID: "0bb79e7b-310f-4013-9d4e-387a6433ba9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.128290 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.126856 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-kube-api-access-gvm46" pod "8a8916eb-d97c-414b-ad8c-03dfda5aebc6" (UID: "8a8916eb-d97c-414b-ad8c-03dfda5aebc6"). InnerVolumeSpecName "kube-api-access-gvm46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.128290 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.127548 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb79e7b-310f-4013-9d4e-387a6433ba9f-kube-api-access-hvd8r" pod "0bb79e7b-310f-4013-9d4e-387a6433ba9f" (UID: "0bb79e7b-310f-4013-9d4e-387a6433ba9f"). InnerVolumeSpecName "kube-api-access-hvd8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.217296 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.217249 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvm46\" (UniqueName: \"kubernetes.io/projected/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-kube-api-access-gvm46\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:19.217410 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.217299 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvd8r\" (UniqueName: \"kubernetes.io/projected/0bb79e7b-310f-4013-9d4e-387a6433ba9f-kube-api-access-hvd8r\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:19.217410 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.217319 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8916eb-d97c-414b-ad8c-03dfda5aebc6-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:19.217410 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.217335 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb79e7b-310f-4013-9d4e-387a6433ba9f-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:19.735823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.735793 4862 generic.go:356] "Generic (PLEG): container finished" podID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerID="690fcdde1284924e2498a484d6a0cfe3a5ebdc1644ff3b4b80b05894956537ce" exitCode=60 Oct 30 19:00:19.735968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.735867 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" event={"ID":"d9f5496e-2536-4aae-a65e-76e2bac7b327","Type":"ContainerDied","Data":"690fcdde1284924e2498a484d6a0cfe3a5ebdc1644ff3b4b80b05894956537ce"} Oct 30 19:00:19.735968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.735937 4862 scope.go:122] "RemoveContainer" containerID="d2a8efd5749b3b24086dbfb0f47c599115d5022031dbcc362b23c68352e0559b" Oct 30 19:00:19.736311 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.736256 4862 scope.go:122] "RemoveContainer" containerID="690fcdde1284924e2498a484d6a0cfe3a5ebdc1644ff3b4b80b05894956537ce" Oct 30 19:00:19.736656 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:19.736630 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr_odh-ai-gateway-infra(d9f5496e-2536-4aae-a65e-76e2bac7b327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" Oct 30 19:00:19.739130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.739109 4862 generic.go:356] "Generic (PLEG): container finished" podID="7d313143-8642-4426-8660-66d58f0702b3" containerID="75bfa8f8ae4ce60adc9d01bd2d271b7704f58eec127ab7c22cccfa19bf6f983e" exitCode=60 Oct 30 19:00:19.739245 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.739174 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" event={"ID":"7d313143-8642-4426-8660-66d58f0702b3","Type":"ContainerDied","Data":"75bfa8f8ae4ce60adc9d01bd2d271b7704f58eec127ab7c22cccfa19bf6f983e"} Oct 30 19:00:19.739723 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.739613 4862 scope.go:122] "RemoveContainer" containerID="75bfa8f8ae4ce60adc9d01bd2d271b7704f58eec127ab7c22cccfa19bf6f983e" Oct 30 19:00:19.740025 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:19.739974 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk_odh-ai-gateway-infra(7d313143-8642-4426-8660-66d58f0702b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" podUID="7d313143-8642-4426-8660-66d58f0702b3" Oct 30 19:00:19.741538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.741517 4862 generic.go:356] "Generic (PLEG): container finished" podID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerID="190016b014c0ad9aa13a38c4d4c30ed8bb5cdf0abe30fb84d9019f56acc0a47f" exitCode=60 Oct 30 19:00:19.741654 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.741593 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" event={"ID":"cfdf45ce-60c7-44a1-91bf-21236823cbff","Type":"ContainerDied","Data":"190016b014c0ad9aa13a38c4d4c30ed8bb5cdf0abe30fb84d9019f56acc0a47f"} Oct 30 19:00:19.741936 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.741921 4862 scope.go:122] "RemoveContainer" containerID="190016b014c0ad9aa13a38c4d4c30ed8bb5cdf0abe30fb84d9019f56acc0a47f" Oct 30 19:00:19.742300 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:19.742240 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w_odh-ai-gateway-infra(cfdf45ce-60c7-44a1-91bf-21236823cbff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" Oct 30 19:00:19.743334 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.743308 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" event={"ID":"8a8916eb-d97c-414b-ad8c-03dfda5aebc6","Type":"ContainerDied","Data":"4552aa6a07ca815e4b8b4abfa5090cde030c1165a0f0aadaee0d18a049d788f5"} Oct 30 19:00:19.743334 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.743329 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d" Oct 30 19:00:19.745112 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.745082 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" event={"ID":"0bb79e7b-310f-4013-9d4e-387a6433ba9f","Type":"ContainerDied","Data":"d3f9c4e55fe8938b82d41b66b37916129b596d6e6a1c334cf7fe07e21aac5b30"} Oct 30 19:00:19.745218 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.745123 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4" Oct 30 19:00:19.755702 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.755689 4862 scope.go:122] "RemoveContainer" containerID="9ad0c83efcb88f8ebe003294409ddf7bacd810ec2f0f3a35ad612985f8131741" Oct 30 19:00:19.773993 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.773975 4862 scope.go:122] "RemoveContainer" containerID="c3080cdac51ecb07f583e6eb47ab97439a23aa947d1c3b9642229b7d388552da" Oct 30 19:00:19.790648 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.790611 4862 scope.go:122] "RemoveContainer" containerID="b56c27b27bbb13eb72105793da3ac76c0d98f0a825a67ac75ad790462b903162" Oct 30 19:00:19.802318 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.802292 4862 scope.go:122] "RemoveContainer" containerID="25d589bdc3c373fef51ef9bac0df41065527a2ecfa31bc487c15592e8e2cdd8a" Oct 30 19:00:19.818489 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.818452 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w"] Oct 30 19:00:19.855775 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.855743 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk"] Oct 30 19:00:19.898348 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:19.898233 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:00:19.909594 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.909576 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:19.938028 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.938001 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d"] Oct 30 19:00:19.943011 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.942992 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8862aa-29889780-z2n8d"] Oct 30 19:00:19.953860 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:19.953836 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr"] Oct 30 19:00:20.019888 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.019864 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4"] Oct 30 19:00:20.025845 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.025815 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d1a330-29889780-dn5k4"] Oct 30 19:00:20.034209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.034179 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38926db9-ad05-4840-86a2-8f066e0de74c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926db9-ad05-4840-86a2-8f066e0de74c-maas-api-service-ca\") pod \"38926db9-ad05-4840-86a2-8f066e0de74c\" (UID: \"38926db9-ad05-4840-86a2-8f066e0de74c\") " Oct 30 19:00:20.034362 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.034326 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38926db9-ad05-4840-86a2-8f066e0de74c-kube-api-access-88rtz\" (UniqueName: \"kubernetes.io/projected/38926db9-ad05-4840-86a2-8f066e0de74c-kube-api-access-88rtz\") pod \"38926db9-ad05-4840-86a2-8f066e0de74c\" (UID: \"38926db9-ad05-4840-86a2-8f066e0de74c\") " Oct 30 19:00:20.035001 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.034978 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38926db9-ad05-4840-86a2-8f066e0de74c-maas-api-service-ca" pod "38926db9-ad05-4840-86a2-8f066e0de74c" (UID: "38926db9-ad05-4840-86a2-8f066e0de74c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:20.035230 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.035197 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38926db9-ad05-4840-86a2-8f066e0de74c-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:20.036906 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.036884 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38926db9-ad05-4840-86a2-8f066e0de74c-kube-api-access-88rtz" pod "38926db9-ad05-4840-86a2-8f066e0de74c" (UID: "38926db9-ad05-4840-86a2-8f066e0de74c"). InnerVolumeSpecName "kube-api-access-88rtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:20.055016 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.054997 4862 scope.go:122] "RemoveContainer" containerID="51f5de507bee191f9253f5277ea597c672f9cdd17233b9e789181d1171b2b5a2" Oct 30 19:00:20.055160 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.055144 4862 scope.go:122] "RemoveContainer" containerID="41d625549bed7d823b8c600b33e822d46892b54cb45c577ff74df1bb7f071116" Oct 30 19:00:20.055329 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.055309 4862 scope.go:122] "RemoveContainer" containerID="bc2aebe69a29d6822a7dcf814092ca99034848a772e19aa5a3cf95108ed3e085" Oct 30 19:00:20.136670 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.136645 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88rtz\" (UniqueName: \"kubernetes.io/projected/38926db9-ad05-4840-86a2-8f066e0de74c-kube-api-access-88rtz\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:20.765116 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.765076 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" Oct 30 19:00:20.765116 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.765085 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz" event={"ID":"38926db9-ad05-4840-86a2-8f066e0de74c","Type":"ContainerDied","Data":"ab9dd2ca9f09888441de2850f60689cf803b6eda89dab66d554ae7df0a478b29"} Oct 30 19:00:20.765446 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.765195 4862 scope.go:122] "RemoveContainer" containerID="bc5c96337dc2919413925608c4880084670b629a5bf9656da2aa2f476b06f9c3" Oct 30 19:00:20.769118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.769082 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" event={"ID":"72239397-4285-47ff-b37f-c42943e13b82","Type":"ContainerStarted","Data":"6d1aaa11e1f4d8af2dae421f646c0cb12dda5337118ab8621fa48232e8beb92d"} Oct 30 19:00:20.773014 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.772987 4862 generic.go:356] "Generic (PLEG): container finished" podID="8888996a-7141-4f3f-83b3-d87811232f6a" containerID="67a2bc97d8c63623b03dc176163803ea824168aca49e9ba34b981b5720a4e8fc" exitCode=60 Oct 30 19:00:20.773103 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.773051 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" event={"ID":"8888996a-7141-4f3f-83b3-d87811232f6a","Type":"ContainerDied","Data":"67a2bc97d8c63623b03dc176163803ea824168aca49e9ba34b981b5720a4e8fc"} Oct 30 19:00:20.773444 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.773427 4862 scope.go:122] "RemoveContainer" containerID="67a2bc97d8c63623b03dc176163803ea824168aca49e9ba34b981b5720a4e8fc" Oct 30 19:00:20.773709 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:20.773691 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2_odh-ai-gateway-infra(8888996a-7141-4f3f-83b3-d87811232f6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" Oct 30 19:00:20.775072 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.775053 4862 generic.go:356] "Generic (PLEG): container finished" podID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerID="0d0b392097c1bcfddfd6c5eb85e0552741881b3e6c7a15ca021c55d269b73cd6" exitCode=60 Oct 30 19:00:20.775152 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.775121 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" event={"ID":"5edcb24a-8fa2-4038-a6a6-778febfd0a7d","Type":"ContainerDied","Data":"0d0b392097c1bcfddfd6c5eb85e0552741881b3e6c7a15ca021c55d269b73cd6"} Oct 30 19:00:20.775464 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.775445 4862 scope.go:122] "RemoveContainer" containerID="0d0b392097c1bcfddfd6c5eb85e0552741881b3e6c7a15ca021c55d269b73cd6" Oct 30 19:00:20.775712 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:20.775694 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz_odh-ai-gateway-infra(5edcb24a-8fa2-4038-a6a6-778febfd0a7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" Oct 30 19:00:20.780789 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.780765 4862 generic.go:356] "Generic (PLEG): container finished" podID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerID="d43dce158afaefb61c73d670418df11399e63a52b49eedd7d4c40a1c1eae876d" exitCode=60 Oct 30 19:00:20.780898 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.780846 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" event={"ID":"1d6fae04-3d28-4cf2-a8d5-a58c9be23015","Type":"ContainerDied","Data":"d43dce158afaefb61c73d670418df11399e63a52b49eedd7d4c40a1c1eae876d"} Oct 30 19:00:20.781200 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.781182 4862 scope.go:122] "RemoveContainer" containerID="d43dce158afaefb61c73d670418df11399e63a52b49eedd7d4c40a1c1eae876d" Oct 30 19:00:20.781519 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:20.781494 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z_odh-ai-gateway-infra(1d6fae04-3d28-4cf2-a8d5-a58c9be23015)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" Oct 30 19:00:20.785724 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.785661 4862 generic.go:356] "Generic (PLEG): container finished" podID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerID="ab9224f464b4569bbaa60bff10d3725c815c4868c42022ab32110e91467afcae" exitCode=60 Oct 30 19:00:20.785819 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.785729 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" event={"ID":"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7","Type":"ContainerDied","Data":"ab9224f464b4569bbaa60bff10d3725c815c4868c42022ab32110e91467afcae"} Oct 30 19:00:20.786134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.786115 4862 scope.go:122] "RemoveContainer" containerID="ab9224f464b4569bbaa60bff10d3725c815c4868c42022ab32110e91467afcae" Oct 30 19:00:20.786463 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:20.786436 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv_odh-ai-gateway-infra(3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" Oct 30 19:00:20.788372 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.788350 4862 generic.go:356] "Generic (PLEG): container finished" podID="1efeda39-fc25-4232-8bba-e023790536cc" containerID="5cfe5a1c2959b55e4e5f802b70f6de56e79e7eddc75661de8146fbfb4a2bd4f1" exitCode=60 Oct 30 19:00:20.788512 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.788419 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" event={"ID":"1efeda39-fc25-4232-8bba-e023790536cc","Type":"ContainerDied","Data":"5cfe5a1c2959b55e4e5f802b70f6de56e79e7eddc75661de8146fbfb4a2bd4f1"} Oct 30 19:00:20.788708 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.788683 4862 scope.go:122] "RemoveContainer" containerID="5cfe5a1c2959b55e4e5f802b70f6de56e79e7eddc75661de8146fbfb4a2bd4f1" Oct 30 19:00:20.788933 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:20.788914 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7_odh-ai-gateway-infra(1efeda39-fc25-4232-8bba-e023790536cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" podUID="1efeda39-fc25-4232-8bba-e023790536cc" Oct 30 19:00:20.793184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.791825 4862 generic.go:356] "Generic (PLEG): container finished" podID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerID="80ff4c58fe4be255d26c1dfe5a5c1a3c1be032289cb85ac5372b42ea4d81a812" exitCode=60 Oct 30 19:00:20.793184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.791936 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" event={"ID":"5704fe4a-6a39-418d-93ef-c5080c48b9a5","Type":"ContainerDied","Data":"80ff4c58fe4be255d26c1dfe5a5c1a3c1be032289cb85ac5372b42ea4d81a812"} Oct 30 19:00:20.793184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.792351 4862 scope.go:122] "RemoveContainer" containerID="80ff4c58fe4be255d26c1dfe5a5c1a3c1be032289cb85ac5372b42ea4d81a812" Oct 30 19:00:20.793184 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:20.792612 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj_odh-ai-gateway-infra(5704fe4a-6a39-418d-93ef-c5080c48b9a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" Oct 30 19:00:20.796059 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.796037 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" event={"ID":"09356c3c-e94c-4a79-855a-fc550e0eb5f1","Type":"ContainerStarted","Data":"6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de"} Oct 30 19:00:20.800353 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.799302 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" event={"ID":"ad42ac0a-7ac9-4996-a585-a77c79873164","Type":"ContainerStarted","Data":"eb1a6345942922e6425b75dafb46a35fb705f6f7b1fcf20f53fa3943719ce8d6"} Oct 30 19:00:20.886292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:20.886235 4862 scope.go:122] "RemoveContainer" containerID="ca2e0f3441c0de309d016d0a9569934a0a221bb8af9953f1967880862d22b863" Oct 30 19:00:21.010308 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.008100 4862 scope.go:122] "RemoveContainer" containerID="9539f51e4a1385fc0b7bc4acc909f449a7d6041b14f9e96f3032bd91a0ea102d" Oct 30 19:00:21.038135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.038050 4862 scope.go:122] "RemoveContainer" containerID="11037ee773c3744b4db3200f3f24d45d7670d00a6fd59cf66870d2612a4e31d3" Oct 30 19:00:21.043592 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.043570 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:21.055847 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.055820 4862 scope.go:122] "RemoveContainer" containerID="3ca943ed72b3107dcceffb73298acceb5e982ec166411982492c56a24656e401" Oct 30 19:00:21.056395 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.056373 4862 scope.go:122] "RemoveContainer" containerID="e09261643d31463f4a1d60d7aa49ec69243bf269c2b63112716763155b176bdc" Oct 30 19:00:21.063785 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.060838 4862 scope.go:122] "RemoveContainer" containerID="97df060a26f2f90c71f7c50123a6bc4ccb60ca46c05e8d9cc9567b7f981c772e" Oct 30 19:00:21.073375 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.073344 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" path="/var/lib/kubelet/pods/0bb79e7b-310f-4013-9d4e-387a6433ba9f/volumes" Oct 30 19:00:21.074038 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.074013 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" path="/var/lib/kubelet/pods/8a8916eb-d97c-414b-ad8c-03dfda5aebc6/volumes" Oct 30 19:00:21.102639 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.102286 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f5496e-2536-4aae-a65e-76e2bac7b327-kube-api-access-srgpp\" (UniqueName: \"kubernetes.io/projected/d9f5496e-2536-4aae-a65e-76e2bac7b327-kube-api-access-srgpp\") pod \"d9f5496e-2536-4aae-a65e-76e2bac7b327\" (UID: \"d9f5496e-2536-4aae-a65e-76e2bac7b327\") " Oct 30 19:00:21.102639 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.102369 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f5496e-2536-4aae-a65e-76e2bac7b327-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5496e-2536-4aae-a65e-76e2bac7b327-maas-api-service-ca\") pod \"d9f5496e-2536-4aae-a65e-76e2bac7b327\" (UID: \"d9f5496e-2536-4aae-a65e-76e2bac7b327\") " Oct 30 19:00:21.103952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.103925 4862 scope.go:122] "RemoveContainer" containerID="de816ecf95c63594bf406941a18ae74a13f9ce9c5341a71d611bf8cbe83b7bb6" Oct 30 19:00:21.104065 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.103989 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f5496e-2536-4aae-a65e-76e2bac7b327-maas-api-service-ca" pod "d9f5496e-2536-4aae-a65e-76e2bac7b327" (UID: "d9f5496e-2536-4aae-a65e-76e2bac7b327"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:21.106599 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.106571 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f5496e-2536-4aae-a65e-76e2bac7b327-kube-api-access-srgpp" pod "d9f5496e-2536-4aae-a65e-76e2bac7b327" (UID: "d9f5496e-2536-4aae-a65e-76e2bac7b327"). InnerVolumeSpecName "kube-api-access-srgpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:21.124909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.124877 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:21.175397 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.175367 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz"] Oct 30 19:00:21.179827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.179805 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6018b4-29889780-hk7bz"] Oct 30 19:00:21.204446 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.204416 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d313143-8642-4426-8660-66d58f0702b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d313143-8642-4426-8660-66d58f0702b3-maas-api-service-ca\") pod \"7d313143-8642-4426-8660-66d58f0702b3\" (UID: \"7d313143-8642-4426-8660-66d58f0702b3\") " Oct 30 19:00:21.205292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.204614 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d313143-8642-4426-8660-66d58f0702b3-kube-api-access-dwknk\" (UniqueName: \"kubernetes.io/projected/7d313143-8642-4426-8660-66d58f0702b3-kube-api-access-dwknk\") pod \"7d313143-8642-4426-8660-66d58f0702b3\" (UID: \"7d313143-8642-4426-8660-66d58f0702b3\") " Oct 30 19:00:21.205292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.204807 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d313143-8642-4426-8660-66d58f0702b3-maas-api-service-ca" pod "7d313143-8642-4426-8660-66d58f0702b3" (UID: "7d313143-8642-4426-8660-66d58f0702b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:21.205292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.204935 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:21.205292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.205229 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d313143-8642-4426-8660-66d58f0702b3-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.205292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.205245 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srgpp\" (UniqueName: \"kubernetes.io/projected/d9f5496e-2536-4aae-a65e-76e2bac7b327-kube-api-access-srgpp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.205292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.205254 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9f5496e-2536-4aae-a65e-76e2bac7b327-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.230580 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.230555 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d313143-8642-4426-8660-66d58f0702b3-kube-api-access-dwknk" pod "7d313143-8642-4426-8660-66d58f0702b3" (UID: "7d313143-8642-4426-8660-66d58f0702b3"). InnerVolumeSpecName "kube-api-access-dwknk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:21.239672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.239649 4862 scope.go:122] "RemoveContainer" containerID="9980d02305bbdd988cbb81c7fd09d41529e0b4017d25243da1719a0c691dd835" Oct 30 19:00:21.306241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.306211 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfdf45ce-60c7-44a1-91bf-21236823cbff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf45ce-60c7-44a1-91bf-21236823cbff-maas-api-service-ca\") pod \"cfdf45ce-60c7-44a1-91bf-21236823cbff\" (UID: \"cfdf45ce-60c7-44a1-91bf-21236823cbff\") " Oct 30 19:00:21.306425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.306250 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfdf45ce-60c7-44a1-91bf-21236823cbff-kube-api-access-wfcrv\" (UniqueName: \"kubernetes.io/projected/cfdf45ce-60c7-44a1-91bf-21236823cbff-kube-api-access-wfcrv\") pod \"cfdf45ce-60c7-44a1-91bf-21236823cbff\" (UID: \"cfdf45ce-60c7-44a1-91bf-21236823cbff\") " Oct 30 19:00:21.306613 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.306590 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfdf45ce-60c7-44a1-91bf-21236823cbff-maas-api-service-ca" pod "cfdf45ce-60c7-44a1-91bf-21236823cbff" (UID: "cfdf45ce-60c7-44a1-91bf-21236823cbff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:21.306922 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.306904 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdf45ce-60c7-44a1-91bf-21236823cbff-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.306995 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.306927 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwknk\" (UniqueName: \"kubernetes.io/projected/7d313143-8642-4426-8660-66d58f0702b3-kube-api-access-dwknk\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.330032 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.329918 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdf45ce-60c7-44a1-91bf-21236823cbff-kube-api-access-wfcrv" pod "cfdf45ce-60c7-44a1-91bf-21236823cbff" (UID: "cfdf45ce-60c7-44a1-91bf-21236823cbff"). InnerVolumeSpecName "kube-api-access-wfcrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:21.408622 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.408586 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfcrv\" (UniqueName: \"kubernetes.io/projected/cfdf45ce-60c7-44a1-91bf-21236823cbff-kube-api-access-wfcrv\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:21.812675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.812636 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" event={"ID":"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8","Type":"ContainerStarted","Data":"03adf74d471b779506b56efc11c69daee20c741f80237369d22cd38110f07fe3"} Oct 30 19:00:21.814878 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.814854 4862 generic.go:356] "Generic (PLEG): container finished" podID="72239397-4285-47ff-b37f-c42943e13b82" containerID="6d1aaa11e1f4d8af2dae421f646c0cb12dda5337118ab8621fa48232e8beb92d" exitCode=60 Oct 30 19:00:21.815001 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.814928 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" event={"ID":"72239397-4285-47ff-b37f-c42943e13b82","Type":"ContainerDied","Data":"6d1aaa11e1f4d8af2dae421f646c0cb12dda5337118ab8621fa48232e8beb92d"} Oct 30 19:00:21.815001 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.814988 4862 scope.go:122] "RemoveContainer" containerID="51f5de507bee191f9253f5277ea597c672f9cdd17233b9e789181d1171b2b5a2" Oct 30 19:00:21.815322 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.815291 4862 scope.go:122] "RemoveContainer" containerID="6d1aaa11e1f4d8af2dae421f646c0cb12dda5337118ab8621fa48232e8beb92d" Oct 30 19:00:21.815577 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:21.815554 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8_odh-ai-gateway-infra(72239397-4285-47ff-b37f-c42943e13b82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" podUID="72239397-4285-47ff-b37f-c42943e13b82" Oct 30 19:00:21.816553 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.816534 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" Oct 30 19:00:21.816695 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.816533 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr" event={"ID":"d9f5496e-2536-4aae-a65e-76e2bac7b327","Type":"ContainerDied","Data":"6717265fba9c8d7f454cbfa7c8c34ae7873f396e64b01b39a5b6cfd4905c0ad3"} Oct 30 19:00:21.821539 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.821519 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" Oct 30 19:00:21.821637 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.821520 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk" event={"ID":"7d313143-8642-4426-8660-66d58f0702b3","Type":"ContainerDied","Data":"8ef1c733c58abe8929cd2ddd1dfcfcc01ddd16338753cfff816daef67bf91257"} Oct 30 19:00:21.823312 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.823292 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" Oct 30 19:00:21.823396 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.823306 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w" event={"ID":"cfdf45ce-60c7-44a1-91bf-21236823cbff","Type":"ContainerDied","Data":"5aeeb96764ab8c6ef44390d9d81c64b03c776505735e4a4e0128a3bb21e62fae"} Oct 30 19:00:21.835496 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.835474 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8"] Oct 30 19:00:21.836065 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.836045 4862 scope.go:122] "RemoveContainer" containerID="690fcdde1284924e2498a484d6a0cfe3a5ebdc1644ff3b4b80b05894956537ce" Oct 30 19:00:21.837972 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.837952 4862 generic.go:356] "Generic (PLEG): container finished" podID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerID="6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de" exitCode=60 Oct 30 19:00:21.838068 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.838019 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" event={"ID":"09356c3c-e94c-4a79-855a-fc550e0eb5f1","Type":"ContainerDied","Data":"6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de"} Oct 30 19:00:21.839465 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.839448 4862 scope.go:122] "RemoveContainer" containerID="6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de" Oct 30 19:00:21.839741 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:21.839720 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv_odh-ai-gateway-infra(09356c3c-e94c-4a79-855a-fc550e0eb5f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" Oct 30 19:00:21.840066 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.840034 4862 generic.go:356] "Generic (PLEG): container finished" podID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerID="eb1a6345942922e6425b75dafb46a35fb705f6f7b1fcf20f53fa3943719ce8d6" exitCode=60 Oct 30 19:00:21.840156 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.840104 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" event={"ID":"ad42ac0a-7ac9-4996-a585-a77c79873164","Type":"ContainerDied","Data":"eb1a6345942922e6425b75dafb46a35fb705f6f7b1fcf20f53fa3943719ce8d6"} Oct 30 19:00:21.840503 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.840485 4862 scope.go:122] "RemoveContainer" containerID="eb1a6345942922e6425b75dafb46a35fb705f6f7b1fcf20f53fa3943719ce8d6" Oct 30 19:00:21.840759 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:21.840740 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp_odh-ai-gateway-infra(ad42ac0a-7ac9-4996-a585-a77c79873164)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" Oct 30 19:00:21.842311 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.842291 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" event={"ID":"b4ea8704-add1-4406-8aa4-d46f42e68614","Type":"ContainerStarted","Data":"4cb9e669b9223e185bff23768043d8e1b97bf4914106dd7658ac8cc96f981379"} Oct 30 19:00:21.848116 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.848097 4862 scope.go:122] "RemoveContainer" containerID="75bfa8f8ae4ce60adc9d01bd2d271b7704f58eec127ab7c22cccfa19bf6f983e" Oct 30 19:00:21.859445 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.859425 4862 scope.go:122] "RemoveContainer" containerID="190016b014c0ad9aa13a38c4d4c30ed8bb5cdf0abe30fb84d9019f56acc0a47f" Oct 30 19:00:21.870242 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.870217 4862 scope.go:122] "RemoveContainer" containerID="41d625549bed7d823b8c600b33e822d46892b54cb45c577ff74df1bb7f071116" Oct 30 19:00:21.884710 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:21.884694 4862 scope.go:122] "RemoveContainer" containerID="bc2aebe69a29d6822a7dcf814092ca99034848a772e19aa5a3cf95108ed3e085" Oct 30 19:00:22.044527 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.044487 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w"] Oct 30 19:00:22.059193 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.059156 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-675070-29889780-96k6w"] Oct 30 19:00:22.062049 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.062026 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz"] Oct 30 19:00:22.092688 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.092660 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk"] Oct 30 19:00:22.104733 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.104702 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-236fc4-29889780-7ftbk"] Oct 30 19:00:22.155523 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.155457 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2"] Oct 30 19:00:22.184826 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.184794 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr"] Oct 30 19:00:22.196599 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.196562 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-861bb3-29889780-v5mkr"] Oct 30 19:00:22.227772 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.227542 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z"] Oct 30 19:00:22.288679 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.288644 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:22.302011 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.299722 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7"] Oct 30 19:00:22.323702 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.323664 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-maas-api-service-ca\") pod \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\" (UID: \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\") " Oct 30 19:00:22.324053 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.324026 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-kube-api-access-22br8\" (UniqueName: \"kubernetes.io/projected/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-kube-api-access-22br8\") pod \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\" (UID: \"5edcb24a-8fa2-4038-a6a6-778febfd0a7d\") " Oct 30 19:00:22.325787 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.325705 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-maas-api-service-ca" pod "5edcb24a-8fa2-4038-a6a6-778febfd0a7d" (UID: "5edcb24a-8fa2-4038-a6a6-778febfd0a7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.333235 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.333213 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv"] Oct 30 19:00:22.350970 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.350219 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-kube-api-access-22br8" pod "5edcb24a-8fa2-4038-a6a6-778febfd0a7d" (UID: "5edcb24a-8fa2-4038-a6a6-778febfd0a7d"). InnerVolumeSpecName "kube-api-access-22br8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.379141 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.370980 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv"] Oct 30 19:00:22.429337 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.429305 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.429585 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.429556 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22br8\" (UniqueName: \"kubernetes.io/projected/5edcb24a-8fa2-4038-a6a6-778febfd0a7d-kube-api-access-22br8\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.439828 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.439522 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj"] Oct 30 19:00:22.495563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.495533 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp"] Oct 30 19:00:22.510547 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.510115 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:22.552175 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.551720 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:22.634939 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.633609 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-kube-api-access-gbw2p\" (UniqueName: \"kubernetes.io/projected/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-kube-api-access-gbw2p\") pod \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\" (UID: \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\") " Oct 30 19:00:22.634939 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.633897 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8888996a-7141-4f3f-83b3-d87811232f6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8888996a-7141-4f3f-83b3-d87811232f6a-maas-api-service-ca\") pod \"8888996a-7141-4f3f-83b3-d87811232f6a\" (UID: \"8888996a-7141-4f3f-83b3-d87811232f6a\") " Oct 30 19:00:22.634939 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.634014 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8888996a-7141-4f3f-83b3-d87811232f6a-kube-api-access-mfzpx\" (UniqueName: \"kubernetes.io/projected/8888996a-7141-4f3f-83b3-d87811232f6a-kube-api-access-mfzpx\") pod \"8888996a-7141-4f3f-83b3-d87811232f6a\" (UID: \"8888996a-7141-4f3f-83b3-d87811232f6a\") " Oct 30 19:00:22.634939 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.634110 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-maas-api-service-ca\") pod \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\" (UID: \"1d6fae04-3d28-4cf2-a8d5-a58c9be23015\") " Oct 30 19:00:22.635545 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.635507 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-maas-api-service-ca" pod "1d6fae04-3d28-4cf2-a8d5-a58c9be23015" (UID: "1d6fae04-3d28-4cf2-a8d5-a58c9be23015"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.636436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.636411 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8888996a-7141-4f3f-83b3-d87811232f6a-maas-api-service-ca" pod "8888996a-7141-4f3f-83b3-d87811232f6a" (UID: "8888996a-7141-4f3f-83b3-d87811232f6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.640721 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.640694 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-kube-api-access-gbw2p" pod "1d6fae04-3d28-4cf2-a8d5-a58c9be23015" (UID: "1d6fae04-3d28-4cf2-a8d5-a58c9be23015"). InnerVolumeSpecName "kube-api-access-gbw2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.641230 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.641206 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:22.643369 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.643347 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8888996a-7141-4f3f-83b3-d87811232f6a-kube-api-access-mfzpx" pod "8888996a-7141-4f3f-83b3-d87811232f6a" (UID: "8888996a-7141-4f3f-83b3-d87811232f6a"). InnerVolumeSpecName "kube-api-access-mfzpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.650769 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.650748 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:22.735780 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.735320 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-kube-api-access-dbl68\" (UniqueName: \"kubernetes.io/projected/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-kube-api-access-dbl68\") pod \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\" (UID: \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\") " Oct 30 19:00:22.737229 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.736071 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1efeda39-fc25-4232-8bba-e023790536cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efeda39-fc25-4232-8bba-e023790536cc-maas-api-service-ca\") pod \"1efeda39-fc25-4232-8bba-e023790536cc\" (UID: \"1efeda39-fc25-4232-8bba-e023790536cc\") " Oct 30 19:00:22.737229 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.736189 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efeda39-fc25-4232-8bba-e023790536cc-kube-api-access-nx9ft\" (UniqueName: \"kubernetes.io/projected/1efeda39-fc25-4232-8bba-e023790536cc-kube-api-access-nx9ft\") pod \"1efeda39-fc25-4232-8bba-e023790536cc\" (UID: \"1efeda39-fc25-4232-8bba-e023790536cc\") " Oct 30 19:00:22.737229 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.736317 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-maas-api-service-ca\") pod \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\" (UID: \"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7\") " Oct 30 19:00:22.737229 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.736974 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbw2p\" (UniqueName: \"kubernetes.io/projected/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-kube-api-access-gbw2p\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.737229 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.736994 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8888996a-7141-4f3f-83b3-d87811232f6a-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.737229 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.737010 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfzpx\" (UniqueName: \"kubernetes.io/projected/8888996a-7141-4f3f-83b3-d87811232f6a-kube-api-access-mfzpx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.737229 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.737025 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6fae04-3d28-4cf2-a8d5-a58c9be23015-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.737861 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.737794 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-maas-api-service-ca" pod "3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" (UID: "3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.738459 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.738440 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efeda39-fc25-4232-8bba-e023790536cc-maas-api-service-ca" pod "1efeda39-fc25-4232-8bba-e023790536cc" (UID: "1efeda39-fc25-4232-8bba-e023790536cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.739392 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.739369 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:22.740710 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.740690 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-kube-api-access-dbl68" pod "3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" (UID: "3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7"). InnerVolumeSpecName "kube-api-access-dbl68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.742573 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.742550 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efeda39-fc25-4232-8bba-e023790536cc-kube-api-access-nx9ft" pod "1efeda39-fc25-4232-8bba-e023790536cc" (UID: "1efeda39-fc25-4232-8bba-e023790536cc"). InnerVolumeSpecName "kube-api-access-nx9ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.748901 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:22.748872 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:22.837584 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.837553 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5704fe4a-6a39-418d-93ef-c5080c48b9a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5704fe4a-6a39-418d-93ef-c5080c48b9a5-maas-api-service-ca\") pod \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\" (UID: \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\") " Oct 30 19:00:22.837753 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.837721 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5704fe4a-6a39-418d-93ef-c5080c48b9a5-kube-api-access-6gqzr\" (UniqueName: \"kubernetes.io/projected/5704fe4a-6a39-418d-93ef-c5080c48b9a5-kube-api-access-6gqzr\") pod \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\" (UID: \"5704fe4a-6a39-418d-93ef-c5080c48b9a5\") " Oct 30 19:00:22.837955 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.837913 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5704fe4a-6a39-418d-93ef-c5080c48b9a5-maas-api-service-ca" pod "5704fe4a-6a39-418d-93ef-c5080c48b9a5" (UID: "5704fe4a-6a39-418d-93ef-c5080c48b9a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:22.838672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.838483 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efeda39-fc25-4232-8bba-e023790536cc-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.838672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.838509 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx9ft\" (UniqueName: \"kubernetes.io/projected/1efeda39-fc25-4232-8bba-e023790536cc-kube-api-access-nx9ft\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.838672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.838524 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5704fe4a-6a39-418d-93ef-c5080c48b9a5-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.838672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.838537 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.838672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.838553 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbl68\" (UniqueName: \"kubernetes.io/projected/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7-kube-api-access-dbl68\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.840145 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.840124 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5704fe4a-6a39-418d-93ef-c5080c48b9a5-kube-api-access-6gqzr" pod "5704fe4a-6a39-418d-93ef-c5080c48b9a5" (UID: "5704fe4a-6a39-418d-93ef-c5080c48b9a5"). InnerVolumeSpecName "kube-api-access-6gqzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:22.854078 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.854054 4862 generic.go:356] "Generic (PLEG): container finished" podID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerID="03adf74d471b779506b56efc11c69daee20c741f80237369d22cd38110f07fe3" exitCode=60 Oct 30 19:00:22.854168 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.854130 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" event={"ID":"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8","Type":"ContainerDied","Data":"03adf74d471b779506b56efc11c69daee20c741f80237369d22cd38110f07fe3"} Oct 30 19:00:22.854208 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.854187 4862 scope.go:122] "RemoveContainer" containerID="e09261643d31463f4a1d60d7aa49ec69243bf269c2b63112716763155b176bdc" Oct 30 19:00:22.854590 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.854574 4862 scope.go:122] "RemoveContainer" containerID="03adf74d471b779506b56efc11c69daee20c741f80237369d22cd38110f07fe3" Oct 30 19:00:22.854941 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:22.854918 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7_odh-ai-gateway-infra(e365121d-8f28-44c0-8b9c-3ce63f3b8fe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" Oct 30 19:00:22.858574 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.858543 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" Oct 30 19:00:22.858574 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.858560 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2" event={"ID":"8888996a-7141-4f3f-83b3-d87811232f6a","Type":"ContainerDied","Data":"73f8ceb7a9b9c5dfa06462e9e5a0ac70c08a098d3c1978347b11511f58ee2cba"} Oct 30 19:00:22.860090 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.860060 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" Oct 30 19:00:22.860183 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.860084 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz" event={"ID":"5edcb24a-8fa2-4038-a6a6-778febfd0a7d","Type":"ContainerDied","Data":"6350d55ca46f1fc407ab38d0b6bbf2c9063739c996d7e319a7dfc3bc6bf34b19"} Oct 30 19:00:22.876847 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.876821 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7"] Oct 30 19:00:22.882359 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.882342 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" Oct 30 19:00:22.882964 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.882409 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z" event={"ID":"1d6fae04-3d28-4cf2-a8d5-a58c9be23015","Type":"ContainerDied","Data":"876a6b5d2e39ef03b6bd60fb25ec7b569af2303de9569298d28d25405d742b9f"} Oct 30 19:00:22.891010 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.890959 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" event={"ID":"3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7","Type":"ContainerDied","Data":"bbf3ab409f764722298c508d6cfe588c751572433a68d4a1ed8c23277cd7656f"} Oct 30 19:00:22.892808 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.892778 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" Oct 30 19:00:22.892893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.892828 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7" event={"ID":"1efeda39-fc25-4232-8bba-e023790536cc","Type":"ContainerDied","Data":"726e37372eb64f74a2c903809b9c0ed9877971f5cdeaec53c63d8fc46de3048c"} Oct 30 19:00:22.893388 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.893317 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv" Oct 30 19:00:22.894232 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.894057 4862 scope.go:122] "RemoveContainer" containerID="67a2bc97d8c63623b03dc176163803ea824168aca49e9ba34b981b5720a4e8fc" Oct 30 19:00:22.895977 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.895951 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" Oct 30 19:00:22.896084 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.895981 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj" event={"ID":"5704fe4a-6a39-418d-93ef-c5080c48b9a5","Type":"ContainerDied","Data":"5d767820c33c2d5a68df5b13693bf8ff6059443bb4a1fc81302c03b592523e9b"} Oct 30 19:00:22.904669 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.904647 4862 scope.go:122] "RemoveContainer" containerID="bd8a3b36052d3b312dd6057f766142301df22f6f03947686ee7a4cda007ebc72" Oct 30 19:00:22.924289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.923561 4862 generic.go:356] "Generic (PLEG): container finished" podID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerID="4cb9e669b9223e185bff23768043d8e1b97bf4914106dd7658ac8cc96f981379" exitCode=60 Oct 30 19:00:22.924289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.923648 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" event={"ID":"b4ea8704-add1-4406-8aa4-d46f42e68614","Type":"ContainerDied","Data":"4cb9e669b9223e185bff23768043d8e1b97bf4914106dd7658ac8cc96f981379"} Oct 30 19:00:22.924289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.924099 4862 scope.go:122] "RemoveContainer" containerID="4cb9e669b9223e185bff23768043d8e1b97bf4914106dd7658ac8cc96f981379" Oct 30 19:00:22.924819 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:22.924792 4862 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9_odh-ai-gateway-infra(b4ea8704-add1-4406-8aa4-d46f42e68614)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" Oct 30 19:00:22.932380 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.932356 4862 scope.go:122] "RemoveContainer" containerID="0d0b392097c1bcfddfd6c5eb85e0552741881b3e6c7a15ca021c55d269b73cd6" Oct 30 19:00:22.963571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.957911 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqzr\" (UniqueName: \"kubernetes.io/projected/5704fe4a-6a39-418d-93ef-c5080c48b9a5-kube-api-access-6gqzr\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:22.963571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.958761 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9"] Oct 30 19:00:23.027842 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:22.989376 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z"] Oct 30 19:00:23.027842 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.021136 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1c4bcd-29889780-fkm2z"] Oct 30 19:00:23.028818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.028419 4862 scope.go:122] "RemoveContainer" containerID="d43dce158afaefb61c73d670418df11399e63a52b49eedd7d4c40a1c1eae876d" Oct 30 19:00:23.028818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.028529 4862 scope.go:122] "RemoveContainer" containerID="2823c4093e1e433eaac651107a0bed1878856fd486c8823b4597594e205b1dc2" Oct 30 19:00:23.029411 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.029242 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz"] Oct 30 19:00:23.034852 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.034829 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fe1374-29889780-j2sjz"] Oct 30 19:00:23.069333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.069188 4862 scope.go:122] "RemoveContainer" containerID="bc83dec156b38eee07df6758b0a9615ba7719f94fb7abb5e744cbdbe8a5a0e51" Oct 30 19:00:23.070836 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.070809 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" path="/var/lib/kubelet/pods/1d6fae04-3d28-4cf2-a8d5-a58c9be23015/volumes" Oct 30 19:00:23.071673 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.071603 4862 scope.go:122] "RemoveContainer" containerID="ab9224f464b4569bbaa60bff10d3725c815c4868c42022ab32110e91467afcae" Oct 30 19:00:23.071853 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.071758 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" path="/var/lib/kubelet/pods/38926db9-ad05-4840-86a2-8f066e0de74c/volumes" Oct 30 19:00:23.072435 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.072408 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" path="/var/lib/kubelet/pods/5edcb24a-8fa2-4038-a6a6-778febfd0a7d/volumes" Oct 30 19:00:23.074319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.074299 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d313143-8642-4426-8660-66d58f0702b3" path="/var/lib/kubelet/pods/7d313143-8642-4426-8660-66d58f0702b3/volumes" Oct 30 19:00:23.075024 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.075004 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" path="/var/lib/kubelet/pods/cfdf45ce-60c7-44a1-91bf-21236823cbff/volumes" Oct 30 19:00:23.075730 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.075712 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" path="/var/lib/kubelet/pods/d9f5496e-2536-4aae-a65e-76e2bac7b327/volumes" Oct 30 19:00:23.076810 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.076791 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2"] Oct 30 19:00:23.081198 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.081174 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6babca-29889780-sstk2"] Oct 30 19:00:23.098488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.098464 4862 scope.go:122] "RemoveContainer" containerID="edcafaa6cdc3866f942af0ff8bf193e36993986aa523385694bd58906409f4fd" Oct 30 19:00:23.099350 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.099320 4862 scope.go:122] "RemoveContainer" containerID="5cfe5a1c2959b55e4e5f802b70f6de56e79e7eddc75661de8146fbfb4a2bd4f1" Oct 30 19:00:23.123512 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.123488 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:23.123832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.123816 4862 scope.go:122] "RemoveContainer" containerID="80ff4c58fe4be255d26c1dfe5a5c1a3c1be032289cb85ac5372b42ea4d81a812" Oct 30 19:00:23.124025 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.124011 4862 scope.go:122] "RemoveContainer" containerID="730b3d4699fe1e12e4030268d9dfc2087e2b417ee749ecfaf599446cf36ae01f" Oct 30 19:00:23.150220 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.150198 4862 scope.go:122] "RemoveContainer" containerID="3ca943ed72b3107dcceffb73298acceb5e982ec166411982492c56a24656e401" Oct 30 19:00:23.153655 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.153630 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt"] Oct 30 19:00:23.207070 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.207043 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv"] Oct 30 19:00:23.214359 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.214333 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-09ef1d-29889780-f98qv"] Oct 30 19:00:23.264896 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.263922 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72239397-4285-47ff-b37f-c42943e13b82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72239397-4285-47ff-b37f-c42943e13b82-maas-api-service-ca\") pod \"72239397-4285-47ff-b37f-c42943e13b82\" (UID: \"72239397-4285-47ff-b37f-c42943e13b82\") " Oct 30 19:00:23.264896 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.264217 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72239397-4285-47ff-b37f-c42943e13b82-kube-api-access-ckxcx\" (UniqueName: \"kubernetes.io/projected/72239397-4285-47ff-b37f-c42943e13b82-kube-api-access-ckxcx\") pod \"72239397-4285-47ff-b37f-c42943e13b82\" (UID: \"72239397-4285-47ff-b37f-c42943e13b82\") " Oct 30 19:00:23.265324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.265282 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72239397-4285-47ff-b37f-c42943e13b82-maas-api-service-ca" pod "72239397-4285-47ff-b37f-c42943e13b82" (UID: "72239397-4285-47ff-b37f-c42943e13b82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:23.266430 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.266402 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72239397-4285-47ff-b37f-c42943e13b82-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:23.296419 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.296386 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72239397-4285-47ff-b37f-c42943e13b82-kube-api-access-ckxcx" pod "72239397-4285-47ff-b37f-c42943e13b82" (UID: "72239397-4285-47ff-b37f-c42943e13b82"). InnerVolumeSpecName "kube-api-access-ckxcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:23.296545 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.296443 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj"] Oct 30 19:00:23.303773 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.303746 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8b8e54-29889780-28ntj"] Oct 30 19:00:23.345143 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.345116 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:23.348425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.348403 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:23.368343 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.368309 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckxcx\" (UniqueName: \"kubernetes.io/projected/72239397-4285-47ff-b37f-c42943e13b82-kube-api-access-ckxcx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:23.403769 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.403740 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7"] Oct 30 19:00:23.410073 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.410043 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-daf3ba-29889780-mczl7"] Oct 30 19:00:23.469642 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.469590 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09356c3c-e94c-4a79-855a-fc550e0eb5f1-kube-api-access-gmfp9\" (UniqueName: \"kubernetes.io/projected/09356c3c-e94c-4a79-855a-fc550e0eb5f1-kube-api-access-gmfp9\") pod \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\" (UID: \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\") " Oct 30 19:00:23.469827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.469651 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09356c3c-e94c-4a79-855a-fc550e0eb5f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09356c3c-e94c-4a79-855a-fc550e0eb5f1-maas-api-service-ca\") pod \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\" (UID: \"09356c3c-e94c-4a79-855a-fc550e0eb5f1\") " Oct 30 19:00:23.469827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.469704 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad42ac0a-7ac9-4996-a585-a77c79873164-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad42ac0a-7ac9-4996-a585-a77c79873164-maas-api-service-ca\") pod \"ad42ac0a-7ac9-4996-a585-a77c79873164\" (UID: \"ad42ac0a-7ac9-4996-a585-a77c79873164\") " Oct 30 19:00:23.470134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.470116 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad42ac0a-7ac9-4996-a585-a77c79873164-kube-api-access-xpq9d\" (UniqueName: \"kubernetes.io/projected/ad42ac0a-7ac9-4996-a585-a77c79873164-kube-api-access-xpq9d\") pod \"ad42ac0a-7ac9-4996-a585-a77c79873164\" (UID: \"ad42ac0a-7ac9-4996-a585-a77c79873164\") " Oct 30 19:00:23.472030 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.471977 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad42ac0a-7ac9-4996-a585-a77c79873164-maas-api-service-ca" pod "ad42ac0a-7ac9-4996-a585-a77c79873164" (UID: "ad42ac0a-7ac9-4996-a585-a77c79873164"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:23.472571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.472550 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09356c3c-e94c-4a79-855a-fc550e0eb5f1-maas-api-service-ca" pod "09356c3c-e94c-4a79-855a-fc550e0eb5f1" (UID: "09356c3c-e94c-4a79-855a-fc550e0eb5f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:23.474170 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.474106 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad42ac0a-7ac9-4996-a585-a77c79873164-kube-api-access-xpq9d" pod "ad42ac0a-7ac9-4996-a585-a77c79873164" (UID: "ad42ac0a-7ac9-4996-a585-a77c79873164"). InnerVolumeSpecName "kube-api-access-xpq9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:23.474885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.474859 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09356c3c-e94c-4a79-855a-fc550e0eb5f1-kube-api-access-gmfp9" pod "09356c3c-e94c-4a79-855a-fc550e0eb5f1" (UID: "09356c3c-e94c-4a79-855a-fc550e0eb5f1"). InnerVolumeSpecName "kube-api-access-gmfp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:23.574771 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.574736 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpq9d\" (UniqueName: \"kubernetes.io/projected/ad42ac0a-7ac9-4996-a585-a77c79873164-kube-api-access-xpq9d\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:23.574771 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.574775 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmfp9\" (UniqueName: \"kubernetes.io/projected/09356c3c-e94c-4a79-855a-fc550e0eb5f1-kube-api-access-gmfp9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:23.575017 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.574791 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09356c3c-e94c-4a79-855a-fc550e0eb5f1-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:23.575017 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.574806 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad42ac0a-7ac9-4996-a585-a77c79873164-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:23.995081 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.995032 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" event={"ID":"09356c3c-e94c-4a79-855a-fc550e0eb5f1","Type":"ContainerDied","Data":"4fa3ef7630c3bd47d0b07e615cdb7bc0722748da6acd5d6fc9b64a7c7964c86f"} Oct 30 19:00:23.995434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.995124 4862 scope.go:122] "RemoveContainer" containerID="6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de" Oct 30 19:00:23.995434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:23.995249 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv" Oct 30 19:00:24.003557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.003526 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" Oct 30 19:00:24.003687 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.003573 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp" event={"ID":"ad42ac0a-7ac9-4996-a585-a77c79873164","Type":"ContainerDied","Data":"5dcff021e6b62073f5db9dc7dd21999f1294bf72dc538e6a443483b5af9029b7"} Oct 30 19:00:24.019415 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.019360 4862 scope.go:122] "RemoveContainer" containerID="eb1a6345942922e6425b75dafb46a35fb705f6f7b1fcf20f53fa3943719ce8d6" Oct 30 19:00:24.019622 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.019600 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" event={"ID":"72239397-4285-47ff-b37f-c42943e13b82","Type":"ContainerDied","Data":"3eb777518c252dcb09aafcd1895a316173db42f74a21d7e85fd1a5652f4c40ba"} Oct 30 19:00:24.019715 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.019693 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8" Oct 30 19:00:24.042329 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.042246 4862 scope.go:122] "RemoveContainer" containerID="6d1aaa11e1f4d8af2dae421f646c0cb12dda5337118ab8621fa48232e8beb92d" Oct 30 19:00:24.091648 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.091619 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv"] Oct 30 19:00:24.098837 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.098802 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv"] Oct 30 19:00:24.171832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.171799 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8"] Oct 30 19:00:24.176083 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.176055 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2107d4-29889780-zfcb8"] Oct 30 19:00:24.203431 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.203370 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp"] Oct 30 19:00:24.208190 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.208164 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:24.209858 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.209650 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-869b55-29889780-fc7zp"] Oct 30 19:00:24.302502 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.302466 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-kube-api-access-sjwbt\" (UniqueName: \"kubernetes.io/projected/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-kube-api-access-sjwbt\") pod \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\" (UID: \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\") " Oct 30 19:00:24.302836 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.302813 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-maas-api-service-ca\") pod \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\" (UID: \"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8\") " Oct 30 19:00:24.304444 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.304420 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-maas-api-service-ca" pod "e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" (UID: "e365121d-8f28-44c0-8b9c-3ce63f3b8fe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:24.308362 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.308323 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-kube-api-access-sjwbt" pod "e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" (UID: "e365121d-8f28-44c0-8b9c-3ce63f3b8fe8"). InnerVolumeSpecName "kube-api-access-sjwbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:24.332011 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.331987 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:24.405014 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.404980 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ea8704-add1-4406-8aa4-d46f42e68614-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea8704-add1-4406-8aa4-d46f42e68614-maas-api-service-ca\") pod \"b4ea8704-add1-4406-8aa4-d46f42e68614\" (UID: \"b4ea8704-add1-4406-8aa4-d46f42e68614\") " Oct 30 19:00:24.405328 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.405307 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ea8704-add1-4406-8aa4-d46f42e68614-maas-api-service-ca" pod "b4ea8704-add1-4406-8aa4-d46f42e68614" (UID: "b4ea8704-add1-4406-8aa4-d46f42e68614"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:24.405395 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.405367 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ea8704-add1-4406-8aa4-d46f42e68614-kube-api-access-j6zn9\" (UniqueName: \"kubernetes.io/projected/b4ea8704-add1-4406-8aa4-d46f42e68614-kube-api-access-j6zn9\") pod \"b4ea8704-add1-4406-8aa4-d46f42e68614\" (UID: \"b4ea8704-add1-4406-8aa4-d46f42e68614\") " Oct 30 19:00:24.406135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.406105 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4ea8704-add1-4406-8aa4-d46f42e68614-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:24.406241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.406149 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjwbt\" (UniqueName: \"kubernetes.io/projected/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-kube-api-access-sjwbt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:24.406241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.406165 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:24.407706 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.407686 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ea8704-add1-4406-8aa4-d46f42e68614-kube-api-access-j6zn9" pod "b4ea8704-add1-4406-8aa4-d46f42e68614" (UID: "b4ea8704-add1-4406-8aa4-d46f42e68614"). InnerVolumeSpecName "kube-api-access-j6zn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:24.509308 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:24.508475 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6zn9\" (UniqueName: \"kubernetes.io/projected/b4ea8704-add1-4406-8aa4-d46f42e68614-kube-api-access-j6zn9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:25.072478 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.072448 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" Oct 30 19:00:25.079524 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.079491 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" Oct 30 19:00:25.086212 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.086182 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" path="/var/lib/kubelet/pods/09356c3c-e94c-4a79-855a-fc550e0eb5f1/volumes" Oct 30 19:00:25.086840 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.086819 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1efeda39-fc25-4232-8bba-e023790536cc" path="/var/lib/kubelet/pods/1efeda39-fc25-4232-8bba-e023790536cc/volumes" Oct 30 19:00:25.087340 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.087322 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" path="/var/lib/kubelet/pods/3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7/volumes" Oct 30 19:00:25.088157 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.088136 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" path="/var/lib/kubelet/pods/5704fe4a-6a39-418d-93ef-c5080c48b9a5/volumes" Oct 30 19:00:25.088626 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.088607 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72239397-4285-47ff-b37f-c42943e13b82" path="/var/lib/kubelet/pods/72239397-4285-47ff-b37f-c42943e13b82/volumes" Oct 30 19:00:25.089012 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.088997 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" path="/var/lib/kubelet/pods/8888996a-7141-4f3f-83b3-d87811232f6a/volumes" Oct 30 19:00:25.089688 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.089667 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" path="/var/lib/kubelet/pods/ad42ac0a-7ac9-4996-a585-a77c79873164/volumes" Oct 30 19:00:25.090475 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.090343 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9" event={"ID":"b4ea8704-add1-4406-8aa4-d46f42e68614","Type":"ContainerDied","Data":"90a9d881bbadf9d4086d227ded2d78985738efe09798603bd7aa48c107ff300f"} Oct 30 19:00:25.090475 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.090375 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7" event={"ID":"e365121d-8f28-44c0-8b9c-3ce63f3b8fe8","Type":"ContainerDied","Data":"26f060098b9431555df4a5d0a082bc032637e31b9c5bcc0553eec4f31169b234"} Oct 30 19:00:25.090475 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.090422 4862 scope.go:122] "RemoveContainer" containerID="4cb9e669b9223e185bff23768043d8e1b97bf4914106dd7658ac8cc96f981379" Oct 30 19:00:25.102155 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.102139 4862 scope.go:122] "RemoveContainer" containerID="03adf74d471b779506b56efc11c69daee20c741f80237369d22cd38110f07fe3" Oct 30 19:00:25.122400 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.122376 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9"] Oct 30 19:00:25.131933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.131910 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-551d0b-29889780-fjtt9"] Oct 30 19:00:25.168387 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.168366 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7"] Oct 30 19:00:25.176338 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:25.176320 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d82e82-29889780-22pw7"] Oct 30 19:00:25.217771 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:25.217729 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:25.218205 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:25.217796 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:25.912205 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:25.912167 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:27.065857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:27.065811 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" path="/var/lib/kubelet/pods/b4ea8704-add1-4406-8aa4-d46f42e68614/volumes" Oct 30 19:00:27.066528 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:27.066505 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" path="/var/lib/kubelet/pods/e365121d-8f28-44c0-8b9c-3ce63f3b8fe8/volumes" Oct 30 19:00:27.151046 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:27.151018 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:00:28.154683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:28.154637 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" podUID="68d21690-359c-495d-b82c-fa0e5a396e36" containerName="payload-processing" containerID="cri-o://d3f42c033dce73b74b5b32cc0247979a9e8a621d9b9474a428fb33ece437d8cb" gracePeriod=25 Oct 30 19:00:31.357721 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:31.357645 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" podUID="68d21690-359c-495d-b82c-fa0e5a396e36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:00:36.002250 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:36.002212 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:37.608720 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:37.608680 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:39.076031 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:39.075996 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr"] Oct 30 19:00:39.076479 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:39.076247 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" containerID="cri-o://2ea677fa962853f52cd7fc857a9fc15b3bfc22a2b3397b922ff996bf159c4c4e" gracePeriod=30 Oct 30 19:00:41.358188 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:41.358148 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" podUID="68d21690-359c-495d-b82c-fa0e5a396e36" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:00:44.293126 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.293086 4862 generic.go:356] "Generic (PLEG): container finished" podID="f1db3a62-9493-4e08-ad00-6099feb138df" containerID="2ea677fa962853f52cd7fc857a9fc15b3bfc22a2b3397b922ff996bf159c4c4e" exitCode=0 Oct 30 19:00:44.293538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.293163 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" event={"ID":"f1db3a62-9493-4e08-ad00-6099feb138df","Type":"ContainerDied","Data":"2ea677fa962853f52cd7fc857a9fc15b3bfc22a2b3397b922ff996bf159c4c4e"} Oct 30 19:00:44.362717 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.362659 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 19:00:44.482732 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.482699 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-credential-socket\") pod \"f1db3a62-9493-4e08-ad00-6099feb138df\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " Oct 30 19:00:44.482889 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.482764 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1db3a62-9493-4e08-ad00-6099feb138df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1db3a62-9493-4e08-ad00-6099feb138df-istiod-ca-cert\") pod \"f1db3a62-9493-4e08-ad00-6099feb138df\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " Oct 30 19:00:44.482889 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.482812 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-kube-api-access-pkprx\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-kube-api-access-pkprx\") pod \"f1db3a62-9493-4e08-ad00-6099feb138df\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " Oct 30 19:00:44.482889 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.482840 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1db3a62-9493-4e08-ad00-6099feb138df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1db3a62-9493-4e08-ad00-6099feb138df-istio-podinfo\") pod \"f1db3a62-9493-4e08-ad00-6099feb138df\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " Oct 30 19:00:44.483032 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.482932 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-envoy\") pod \"f1db3a62-9493-4e08-ad00-6099feb138df\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " Oct 30 19:00:44.483032 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.482972 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-credential-socket" pod "f1db3a62-9493-4e08-ad00-6099feb138df" (UID: "f1db3a62-9493-4e08-ad00-6099feb138df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:44.483119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.483027 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-certs\") pod \"f1db3a62-9493-4e08-ad00-6099feb138df\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " Oct 30 19:00:44.483119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.483060 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-data\") pod \"f1db3a62-9493-4e08-ad00-6099feb138df\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " Oct 30 19:00:44.483234 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.483139 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-socket\") pod \"f1db3a62-9493-4e08-ad00-6099feb138df\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " Oct 30 19:00:44.483234 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.483201 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1db3a62-9493-4e08-ad00-6099feb138df-istiod-ca-cert" pod "f1db3a62-9493-4e08-ad00-6099feb138df" (UID: "f1db3a62-9493-4e08-ad00-6099feb138df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:44.483358 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.483236 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-istio-token\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-istio-token\") pod \"f1db3a62-9493-4e08-ad00-6099feb138df\" (UID: \"f1db3a62-9493-4e08-ad00-6099feb138df\") " Oct 30 19:00:44.483417 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.483369 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-certs" pod "f1db3a62-9493-4e08-ad00-6099feb138df" (UID: "f1db3a62-9493-4e08-ad00-6099feb138df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:44.483530 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.483506 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-data" pod "f1db3a62-9493-4e08-ad00-6099feb138df" (UID: "f1db3a62-9493-4e08-ad00-6099feb138df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:44.483632 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.483571 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-socket" pod "f1db3a62-9493-4e08-ad00-6099feb138df" (UID: "f1db3a62-9493-4e08-ad00-6099feb138df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:44.484182 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.484155 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:44.484334 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.484190 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1db3a62-9493-4e08-ad00-6099feb138df-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:44.484334 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.484218 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:44.484334 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.484230 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:44.484334 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.484242 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:44.485365 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.485343 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-envoy" pod "f1db3a62-9493-4e08-ad00-6099feb138df" (UID: "f1db3a62-9493-4e08-ad00-6099feb138df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:44.485540 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.485524 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-kube-api-access-pkprx" pod "f1db3a62-9493-4e08-ad00-6099feb138df" (UID: "f1db3a62-9493-4e08-ad00-6099feb138df"). InnerVolumeSpecName "kube-api-access-pkprx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:44.485833 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.485805 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1db3a62-9493-4e08-ad00-6099feb138df-istio-podinfo" pod "f1db3a62-9493-4e08-ad00-6099feb138df" (UID: "f1db3a62-9493-4e08-ad00-6099feb138df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:00:44.485833 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.485828 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-istio-token" pod "f1db3a62-9493-4e08-ad00-6099feb138df" (UID: "f1db3a62-9493-4e08-ad00-6099feb138df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:44.586094 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.586062 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1db3a62-9493-4e08-ad00-6099feb138df-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:44.586094 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.586089 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:44.586094 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.586098 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkprx\" (UniqueName: \"kubernetes.io/projected/f1db3a62-9493-4e08-ad00-6099feb138df-kube-api-access-pkprx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:44.586347 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:44.586108 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1db3a62-9493-4e08-ad00-6099feb138df-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:45.302336 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:45.302307 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" Oct 30 19:00:45.302726 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:45.302344 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr" event={"ID":"f1db3a62-9493-4e08-ad00-6099feb138df","Type":"ContainerDied","Data":"4161aa7fae8603d557c1b6810613d213b35304074eee221f43064f8950e3b283"} Oct 30 19:00:45.302726 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:45.302406 4862 scope.go:122] "RemoveContainer" containerID="2ea677fa962853f52cd7fc857a9fc15b3bfc22a2b3397b922ff996bf159c4c4e" Oct 30 19:00:45.361049 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:45.360994 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr"] Oct 30 19:00:45.370766 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:45.370740 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-861bb3-openshift-default-547b4c86df-f9ggr"] Oct 30 19:00:46.096566 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:46.096528 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:47.063765 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:47.063731 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" path="/var/lib/kubelet/pods/f1db3a62-9493-4e08-ad00-6099feb138df/volumes" Oct 30 19:00:48.330627 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:48.330590 4862 generic.go:356] "Generic (PLEG): container finished" podID="68d21690-359c-495d-b82c-fa0e5a396e36" containerID="d3f42c033dce73b74b5b32cc0247979a9e8a621d9b9474a428fb33ece437d8cb" exitCode=0 Oct 30 19:00:48.331034 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:48.330659 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" event={"ID":"68d21690-359c-495d-b82c-fa0e5a396e36","Type":"ContainerDied","Data":"d3f42c033dce73b74b5b32cc0247979a9e8a621d9b9474a428fb33ece437d8cb"} Oct 30 19:00:48.441302 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:48.441251 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 19:00:48.553764 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:48.553730 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d21690-359c-495d-b82c-fa0e5a396e36-kube-api-access-7xv6c\" (UniqueName: \"kubernetes.io/projected/68d21690-359c-495d-b82c-fa0e5a396e36-kube-api-access-7xv6c\") pod \"68d21690-359c-495d-b82c-fa0e5a396e36\" (UID: \"68d21690-359c-495d-b82c-fa0e5a396e36\") " Oct 30 19:00:48.553985 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:48.553933 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d21690-359c-495d-b82c-fa0e5a396e36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d21690-359c-495d-b82c-fa0e5a396e36-plugins-config-volume\") pod \"68d21690-359c-495d-b82c-fa0e5a396e36\" (UID: \"68d21690-359c-495d-b82c-fa0e5a396e36\") " Oct 30 19:00:48.554458 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:48.554401 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d21690-359c-495d-b82c-fa0e5a396e36-plugins-config-volume" pod "68d21690-359c-495d-b82c-fa0e5a396e36" (UID: "68d21690-359c-495d-b82c-fa0e5a396e36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:48.554731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:48.554710 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68d21690-359c-495d-b82c-fa0e5a396e36-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:48.565285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:48.564434 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d21690-359c-495d-b82c-fa0e5a396e36-kube-api-access-7xv6c" pod "68d21690-359c-495d-b82c-fa0e5a396e36" (UID: "68d21690-359c-495d-b82c-fa0e5a396e36"). InnerVolumeSpecName "kube-api-access-7xv6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:48.656230 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:48.656201 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xv6c\" (UniqueName: \"kubernetes.io/projected/68d21690-359c-495d-b82c-fa0e5a396e36-kube-api-access-7xv6c\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:00:49.340060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:49.339982 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" Oct 30 19:00:49.340060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:49.339995 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt" event={"ID":"68d21690-359c-495d-b82c-fa0e5a396e36","Type":"ContainerDied","Data":"99ec11ac1e4f3a4e0b874e5d65853c0b6e8999c97ee00f412abe7a20303ab5a8"} Oct 30 19:00:49.340559 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:49.340073 4862 scope.go:122] "RemoveContainer" containerID="d3f42c033dce73b74b5b32cc0247979a9e8a621d9b9474a428fb33ece437d8cb" Oct 30 19:00:49.387835 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:49.387810 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt"] Oct 30 19:00:49.398378 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:49.398356 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1aa59a-5f69877548-fppdt"] Oct 30 19:00:49.871276 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:49.871230 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:00:51.064169 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:00:51.064134 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d21690-359c-495d-b82c-fa0e5a396e36" path="/var/lib/kubelet/pods/68d21690-359c-495d-b82c-fa0e5a396e36/volumes" Oct 30 19:00:52.682903 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:52.682871 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:00:56.112603 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:00:56.112566 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:01:00.821491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:00.821460 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l"] Oct 30 19:01:06.202161 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:06.202118 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:01:07.621169 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:07.621132 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:01:11.923286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:11.916240 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct"] Oct 30 19:01:11.966047 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:11.965154 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k"] Oct 30 19:01:15.847613 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:15.847568 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" podUID="9f4044a4-d3d1-4a5d-9861-91611fddfb8d" containerName="main" containerID="cri-o://4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0" gracePeriod=45 Oct 30 19:01:16.142584 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.142556 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 19:01:16.231406 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.231373 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-kube-api-access-6blgq\" (UniqueName: \"kubernetes.io/projected/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-kube-api-access-6blgq\") pod \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " Oct 30 19:01:16.231645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.231620 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-home\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-home\") pod \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " Oct 30 19:01:16.231738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.231667 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-model-cache\") pod \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " Oct 30 19:01:16.231738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.231697 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-dshm\") pod \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " Oct 30 19:01:16.231864 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.231750 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tls-certs\") pod \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " Oct 30 19:01:16.231928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.231865 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-home" pod "9f4044a4-d3d1-4a5d-9861-91611fddfb8d" (UID: "9f4044a4-d3d1-4a5d-9861-91611fddfb8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:16.231928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.231915 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tmp-dir\") pod \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\" (UID: \"9f4044a4-d3d1-4a5d-9861-91611fddfb8d\") " Oct 30 19:01:16.232053 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.231985 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-model-cache" pod "9f4044a4-d3d1-4a5d-9861-91611fddfb8d" (UID: "9f4044a4-d3d1-4a5d-9861-91611fddfb8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:16.232861 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.232252 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tmp-dir" pod "9f4044a4-d3d1-4a5d-9861-91611fddfb8d" (UID: "9f4044a4-d3d1-4a5d-9861-91611fddfb8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:16.233010 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.232982 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:16.233010 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.233007 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:16.233195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.233021 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:16.234884 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.234851 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-dshm" pod "9f4044a4-d3d1-4a5d-9861-91611fddfb8d" (UID: "9f4044a4-d3d1-4a5d-9861-91611fddfb8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:16.234884 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.234861 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tls-certs" pod "9f4044a4-d3d1-4a5d-9861-91611fddfb8d" (UID: "9f4044a4-d3d1-4a5d-9861-91611fddfb8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:16.235078 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.234930 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-kube-api-access-6blgq" pod "9f4044a4-d3d1-4a5d-9861-91611fddfb8d" (UID: "9f4044a4-d3d1-4a5d-9861-91611fddfb8d"). InnerVolumeSpecName "kube-api-access-6blgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:16.302059 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:16.302016 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09356c3c_e94c_4a79_855a_fc550e0eb5f1.slice/crio-conmon-6e7c82872087e5e61eaf59a5c0b5513251a306d2eef979c032ef94f3789925de.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:01:16.335104 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.335065 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6blgq\" (UniqueName: \"kubernetes.io/projected/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-kube-api-access-6blgq\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:16.335104 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.335102 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:16.335344 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.335118 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4044a4-d3d1-4a5d-9861-91611fddfb8d-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:16.603063 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.603006 4862 generic.go:356] "Generic (PLEG): container finished" podID="9f4044a4-d3d1-4a5d-9861-91611fddfb8d" containerID="4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0" exitCode=0 Oct 30 19:01:16.603251 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.603092 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" event={"ID":"9f4044a4-d3d1-4a5d-9861-91611fddfb8d","Type":"ContainerDied","Data":"4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0"} Oct 30 19:01:16.603251 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.603124 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" Oct 30 19:01:16.603251 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.603141 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l" event={"ID":"9f4044a4-d3d1-4a5d-9861-91611fddfb8d","Type":"ContainerDied","Data":"face1f2839caffaf6d56fa38e2b7d6b421c694f3669046d9808bb0a9ca47dbb0"} Oct 30 19:01:16.603251 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.603160 4862 scope.go:122] "RemoveContainer" containerID="4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0" Oct 30 19:01:16.615864 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.615844 4862 scope.go:122] "RemoveContainer" containerID="4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0" Oct 30 19:01:16.616171 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:16.616151 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0\": container with ID starting with 4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0 not found: ID does not exist" containerID="4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0" Oct 30 19:01:16.616319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.616178 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0"} err="failed to get container status \"4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0\": rpc error: code = NotFound desc = could not find container \"4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0\": container with ID starting with 4354d213dfce80863296d6c740cbfed1afb4e1438f6e5b97f03903501f9119d0 not found: ID does not exist" Oct 30 19:01:16.689529 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.689489 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l"] Oct 30 19:01:16.701366 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.701337 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f37ca7/test-model-f37ca7-kserve-5979d9f6df-fxf5l"] Oct 30 19:01:16.916707 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.916608 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" podUID="adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61" containerName="payload-processing" containerID="cri-o://65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775" gracePeriod=25 Oct 30 19:01:16.966094 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:16.966054 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" podUID="a0c69a88-80c5-4d22-a1b3-172e2f6c171d" containerName="payload-pre-processing" containerID="cri-o://66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb" gracePeriod=25 Oct 30 19:01:17.065396 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.065364 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4044a4-d3d1-4a5d-9861-91611fddfb8d" path="/var/lib/kubelet/pods/9f4044a4-d3d1-4a5d-9861-91611fddfb8d/volumes" Oct 30 19:01:17.250348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.250325 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 19:01:17.359958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.359928 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-kube-api-access-8h8m8\" (UniqueName: \"kubernetes.io/projected/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-kube-api-access-8h8m8\") pod \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\" (UID: \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\") " Oct 30 19:01:17.360147 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.360058 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-plugins-config-volume\") pod \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\" (UID: \"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61\") " Oct 30 19:01:17.360477 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.360455 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-plugins-config-volume" pod "adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61" (UID: "adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:17.360718 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.360703 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:17.362305 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.362256 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-kube-api-access-8h8m8" pod "adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61" (UID: "adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61"). InnerVolumeSpecName "kube-api-access-8h8m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:17.462848 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.462761 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h8m8\" (UniqueName: \"kubernetes.io/projected/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61-kube-api-access-8h8m8\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:17.663463 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.663420 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2"] Oct 30 19:01:17.688286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.663695 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" containerID="cri-o://ae43945ff615dd128905956d20fbefea21a5fb7359b9ab741315671641875338" gracePeriod=30 Oct 30 19:01:17.716772 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.716686 4862 generic.go:356] "Generic (PLEG): container finished" podID="adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61" containerID="65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775" exitCode=0 Oct 30 19:01:17.717176 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.717148 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" event={"ID":"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61","Type":"ContainerDied","Data":"65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775"} Oct 30 19:01:17.717381 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.717364 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" event={"ID":"adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61","Type":"ContainerDied","Data":"dc0d72652e7a21eadd4cb8987f9774446894bbe0c57763dbb2cf2f3efb47feb8"} Oct 30 19:01:17.717521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.717507 4862 scope.go:122] "RemoveContainer" containerID="65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775" Oct 30 19:01:17.717794 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.717740 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct" Oct 30 19:01:17.767126 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.766708 4862 scope.go:122] "RemoveContainer" containerID="65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775" Oct 30 19:01:17.767126 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:17.767033 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775\": container with ID starting with 65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775 not found: ID does not exist" containerID="65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775" Oct 30 19:01:17.767126 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.767063 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775"} err="failed to get container status \"65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775\": rpc error: code = NotFound desc = could not find container \"65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775\": container with ID starting with 65b8b2764debcc7e2ce4238f9c85615408e7378a7267114adcbbbecbb47b9775 not found: ID does not exist" Oct 30 19:01:17.848486 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.848451 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct"] Oct 30 19:01:17.870292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:17.870226 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8862aa-55748dd5d8-slvct"] Oct 30 19:01:18.990121 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:18.990077 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" podUID="a0c69a88-80c5-4d22-a1b3-172e2f6c171d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:01:19.077618 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:19.077577 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61" path="/var/lib/kubelet/pods/adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61/volumes" Oct 30 19:01:19.879558 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:19.879525 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:01:21.172774 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:21.172740 4862 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/97f3d614ef013696265e513cd598af72981b176be8aa1559cc085d85566411f7/diff" to get inode usage: stat /var/lib/containers/storage/overlay/97f3d614ef013696265e513cd598af72981b176be8aa1559cc085d85566411f7/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv_09356c3c-e94c-4a79-855a-fc550e0eb5f1/cleanup/2.log" to get inode usage: stat /var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-extmod-a-713dc8-29889780-7mrzv_09356c3c-e94c-4a79-855a-fc550e0eb5f1/cleanup/2.log: no such file or directory Oct 30 19:01:22.979760 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:22.979703 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 19:01:23.095726 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.095681 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-envoy\") pod \"4e97e48f-6b9d-4832-9876-18c1a03d431a\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " Oct 30 19:01:23.095907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.095777 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-socket\") pod \"4e97e48f-6b9d-4832-9876-18c1a03d431a\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " Oct 30 19:01:23.095907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.095841 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-token\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-token\") pod \"4e97e48f-6b9d-4832-9876-18c1a03d431a\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " Oct 30 19:01:23.095907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.095898 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-podinfo\") pod \"4e97e48f-6b9d-4832-9876-18c1a03d431a\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " Oct 30 19:01:23.096087 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.095927 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-credential-socket\") pod \"4e97e48f-6b9d-4832-9876-18c1a03d431a\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " Oct 30 19:01:23.096087 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.095979 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-certs\") pod \"4e97e48f-6b9d-4832-9876-18c1a03d431a\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " Oct 30 19:01:23.096087 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.096040 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-data\") pod \"4e97e48f-6b9d-4832-9876-18c1a03d431a\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " Oct 30 19:01:23.096087 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.096063 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-socket" pod "4e97e48f-6b9d-4832-9876-18c1a03d431a" (UID: "4e97e48f-6b9d-4832-9876-18c1a03d431a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:23.096355 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.096155 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e97e48f-6b9d-4832-9876-18c1a03d431a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e97e48f-6b9d-4832-9876-18c1a03d431a-istiod-ca-cert\") pod \"4e97e48f-6b9d-4832-9876-18c1a03d431a\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " Oct 30 19:01:23.096355 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.096246 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-kube-api-access-dckct\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-kube-api-access-dckct\") pod \"4e97e48f-6b9d-4832-9876-18c1a03d431a\" (UID: \"4e97e48f-6b9d-4832-9876-18c1a03d431a\") " Oct 30 19:01:23.096355 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.096305 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-credential-socket" pod "4e97e48f-6b9d-4832-9876-18c1a03d431a" (UID: "4e97e48f-6b9d-4832-9876-18c1a03d431a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:23.096505 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.096351 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-data" pod "4e97e48f-6b9d-4832-9876-18c1a03d431a" (UID: "4e97e48f-6b9d-4832-9876-18c1a03d431a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:23.096732 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.096706 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e97e48f-6b9d-4832-9876-18c1a03d431a-istiod-ca-cert" pod "4e97e48f-6b9d-4832-9876-18c1a03d431a" (UID: "4e97e48f-6b9d-4832-9876-18c1a03d431a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:23.097346 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.096973 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-certs" pod "4e97e48f-6b9d-4832-9876-18c1a03d431a" (UID: "4e97e48f-6b9d-4832-9876-18c1a03d431a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:23.097346 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.097333 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:23.097500 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.097357 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:23.097500 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.097373 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:23.097500 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.097388 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:23.097500 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.097400 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e97e48f-6b9d-4832-9876-18c1a03d431a-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:23.098330 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.098302 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-podinfo" pod "4e97e48f-6b9d-4832-9876-18c1a03d431a" (UID: "4e97e48f-6b9d-4832-9876-18c1a03d431a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:01:23.098435 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.098345 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-token" pod "4e97e48f-6b9d-4832-9876-18c1a03d431a" (UID: "4e97e48f-6b9d-4832-9876-18c1a03d431a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:23.098435 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.098377 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-envoy" pod "4e97e48f-6b9d-4832-9876-18c1a03d431a" (UID: "4e97e48f-6b9d-4832-9876-18c1a03d431a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:23.098747 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.098727 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-kube-api-access-dckct" pod "4e97e48f-6b9d-4832-9876-18c1a03d431a" (UID: "4e97e48f-6b9d-4832-9876-18c1a03d431a"). InnerVolumeSpecName "kube-api-access-dckct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:23.198979 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.198948 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:23.198979 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.198982 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:23.199246 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.198996 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e97e48f-6b9d-4832-9876-18c1a03d431a-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:23.199246 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.199011 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dckct\" (UniqueName: \"kubernetes.io/projected/4e97e48f-6b9d-4832-9876-18c1a03d431a-kube-api-access-dckct\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:23.395038 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.395004 4862 scope.go:122] "RemoveContainer" containerID="d1a9c0e4430e76c95d953e88a5d8cbcfcfa6c48dd84c1555262c2f1ed8c61b10" Oct 30 19:01:23.407397 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.407374 4862 scope.go:122] "RemoveContainer" containerID="ae43945ff615dd128905956d20fbefea21a5fb7359b9ab741315671641875338" Oct 30 19:01:23.418104 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.418087 4862 scope.go:122] "RemoveContainer" containerID="4379a7548d64d079d6b1e79dcae1c0e7cb3e0965e94bdf62df12c08547456c07" Oct 30 19:01:23.794195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.794111 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" event={"ID":"4e97e48f-6b9d-4832-9876-18c1a03d431a","Type":"ContainerDied","Data":"ae43945ff615dd128905956d20fbefea21a5fb7359b9ab741315671641875338"} Oct 30 19:01:23.794195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.794169 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" event={"ID":"4e97e48f-6b9d-4832-9876-18c1a03d431a","Type":"ContainerDied","Data":"02e856d4441fbd00f87706c51aa99aa07316233171b185b7fdc10298a88c0e6e"} Oct 30 19:01:23.794195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.794178 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2" Oct 30 19:01:23.916797 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.916751 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2"] Oct 30 19:01:23.925865 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:23.925842 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1aa59a-openshift-default-579999c475-nfnl2"] Oct 30 19:01:25.075213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:25.075176 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" path="/var/lib/kubelet/pods/4e97e48f-6b9d-4832-9876-18c1a03d431a/volumes" Oct 30 19:01:28.397169 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397136 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft"] Oct 30 19:01:28.397817 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397801 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerName="cleanup" Oct 30 19:01:28.397817 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397817 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397829 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397835 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397840 4862 cpu_manager.go:414] "containerMap: removing container" podUID="72239397-4285-47ff-b37f-c42943e13b82" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397846 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72239397-4285-47ff-b37f-c42943e13b82" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397855 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1efeda39-fc25-4232-8bba-e023790536cc" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397859 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efeda39-fc25-4232-8bba-e023790536cc" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397867 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1efeda39-fc25-4232-8bba-e023790536cc" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397871 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efeda39-fc25-4232-8bba-e023790536cc" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397878 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397882 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397890 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397894 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397900 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397904 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397915 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397919 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397926 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397930 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397940 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397944 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397953 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerName="cleanup" Oct 30 19:01:28.397952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397958 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397964 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397969 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397978 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397983 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397988 4862 cpu_manager.go:414] "containerMap: removing container" podUID="7d313143-8642-4426-8660-66d58f0702b3" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.397993 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d313143-8642-4426-8660-66d58f0702b3" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398000 4862 cpu_manager.go:414] "containerMap: removing container" podUID="68d21690-359c-495d-b82c-fa0e5a396e36" containerName="payload-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398004 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d21690-359c-495d-b82c-fa0e5a396e36" containerName="payload-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398010 4862 cpu_manager.go:414] "containerMap: removing container" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398014 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398020 4862 cpu_manager.go:414] "containerMap: removing container" podUID="7d313143-8642-4426-8660-66d58f0702b3" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398024 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d313143-8642-4426-8660-66d58f0702b3" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398030 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1efeda39-fc25-4232-8bba-e023790536cc" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398033 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efeda39-fc25-4232-8bba-e023790536cc" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398042 4862 cpu_manager.go:414] "containerMap: removing container" podUID="adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61" containerName="payload-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398047 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61" containerName="payload-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398053 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398057 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398064 4862 cpu_manager.go:414] "containerMap: removing container" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398068 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398074 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398078 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398083 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398087 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398093 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398096 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398103 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398107 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398113 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398117 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398123 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398127 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398142 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398146 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398152 4862 cpu_manager.go:414] "containerMap: removing container" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398156 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398163 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398166 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398176 4862 cpu_manager.go:414] "containerMap: removing container" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398180 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398187 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398191 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398199 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398203 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398213 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6e82685b-9c65-432b-a863-c39edd6950e4" containerName="payload-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398219 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e82685b-9c65-432b-a863-c39edd6950e4" containerName="payload-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398227 4862 cpu_manager.go:414] "containerMap: removing container" podUID="7d313143-8642-4426-8660-66d58f0702b3" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398231 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d313143-8642-4426-8660-66d58f0702b3" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398237 4862 cpu_manager.go:414] "containerMap: removing container" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398242 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398248 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398253 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398271 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398276 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398282 4862 cpu_manager.go:414] "containerMap: removing container" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398286 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398295 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398299 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398307 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398311 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398320 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398324 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398330 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398335 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398344 4862 cpu_manager.go:414] "containerMap: removing container" podUID="abd590d7-b8b0-4356-9566-c3e62499c87b" containerName="payload-pre-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398348 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd590d7-b8b0-4356-9566-c3e62499c87b" containerName="payload-pre-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398357 4862 cpu_manager.go:414] "containerMap: removing container" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398361 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398369 4862 cpu_manager.go:414] "containerMap: removing container" podUID="72239397-4285-47ff-b37f-c42943e13b82" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398375 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72239397-4285-47ff-b37f-c42943e13b82" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398380 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398385 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398392 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398397 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398408 4862 cpu_manager.go:414] "containerMap: removing container" podUID="9f4044a4-d3d1-4a5d-9861-91611fddfb8d" containerName="main" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398412 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4044a4-d3d1-4a5d-9861-91611fddfb8d" containerName="main" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398419 4862 cpu_manager.go:414] "containerMap: removing container" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398423 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398429 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398433 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398531 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd590d7-b8b0-4356-9566-c3e62499c87b" containerName="payload-pre-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398542 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398548 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="72239397-4285-47ff-b37f-c42943e13b82" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398554 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398560 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398565 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398571 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4044a4-d3d1-4a5d-9861-91611fddfb8d" containerName="main" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398578 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398584 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398589 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398593 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398599 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="72239397-4285-47ff-b37f-c42943e13b82" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398604 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efeda39-fc25-4232-8bba-e023790536cc" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398610 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efeda39-fc25-4232-8bba-e023790536cc" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398616 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398620 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398625 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398629 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398633 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398639 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398644 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398650 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1db3a62-9493-4e08-ad00-6099feb138df" containerName="istio-proxy" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398656 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d313143-8642-4426-8660-66d58f0702b3" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398661 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d21690-359c-495d-b82c-fa0e5a396e36" containerName="payload-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398669 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398674 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d313143-8642-4426-8660-66d58f0702b3" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398678 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efeda39-fc25-4232-8bba-e023790536cc" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398683 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="adcd772e-bd3a-45f6-a8a0-b1bfb97d9b61" containerName="payload-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398688 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="09356c3c-e94c-4a79-855a-fc550e0eb5f1" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398695 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398700 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398705 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398709 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398716 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398723 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398728 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398733 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398739 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398745 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e82685b-9c65-432b-a863-c39edd6950e4" containerName="payload-processing" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398752 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="38926db9-ad05-4840-86a2-8f066e0de74c" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398758 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398764 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f5496e-2536-4aae-a65e-76e2bac7b327" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398770 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398774 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398778 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerName="cleanup" Oct 30 19:01:28.398727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.398784 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerName="cleanup" Oct 30 19:01:28.402747 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.401608 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.457138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.457103 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft"] Oct 30 19:01:28.463725 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.463702 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4"] Oct 30 19:01:28.464657 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464634 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerName="cleanup" Oct 30 19:01:28.464657 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464658 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerName="cleanup" Oct 30 19:01:28.464827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464678 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerName="cleanup" Oct 30 19:01:28.464827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464685 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerName="cleanup" Oct 30 19:01:28.464827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464702 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerName="cleanup" Oct 30 19:01:28.464827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464709 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerName="cleanup" Oct 30 19:01:28.464827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464734 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerName="cleanup" Oct 30 19:01:28.464827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464742 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5704fe4a-6a39-418d-93ef-c5080c48b9a5" containerName="cleanup" Oct 30 19:01:28.464827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464787 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerName="cleanup" Oct 30 19:01:28.464827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464796 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerName="cleanup" Oct 30 19:01:28.465079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464851 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerName="cleanup" Oct 30 19:01:28.465079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464860 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerName="cleanup" Oct 30 19:01:28.465079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464869 4862 cpu_manager.go:414] "containerMap: removing container" podUID="72239397-4285-47ff-b37f-c42943e13b82" containerName="cleanup" Oct 30 19:01:28.465079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.464875 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72239397-4285-47ff-b37f-c42943e13b82" containerName="cleanup" Oct 30 19:01:28.465079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465024 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d313143-8642-4426-8660-66d58f0702b3" containerName="cleanup" Oct 30 19:01:28.465079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465041 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edcb24a-8fa2-4038-a6a6-778febfd0a7d" containerName="cleanup" Oct 30 19:01:28.465079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465058 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8d3f67-cc74-461e-b4a8-4f451f1bdfd7" containerName="cleanup" Oct 30 19:01:28.465079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465066 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6fae04-3d28-4cf2-a8d5-a58c9be23015" containerName="cleanup" Oct 30 19:01:28.465079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465075 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb79e7b-310f-4013-9d4e-387a6433ba9f" containerName="cleanup" Oct 30 19:01:28.465491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465085 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e97e48f-6b9d-4832-9876-18c1a03d431a" containerName="istio-proxy" Oct 30 19:01:28.465491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465095 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8916eb-d97c-414b-ad8c-03dfda5aebc6" containerName="cleanup" Oct 30 19:01:28.465491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465111 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e365121d-8f28-44c0-8b9c-3ce63f3b8fe8" containerName="cleanup" Oct 30 19:01:28.465491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465121 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ea8704-add1-4406-8aa4-d46f42e68614" containerName="cleanup" Oct 30 19:01:28.465491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465130 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="72239397-4285-47ff-b37f-c42943e13b82" containerName="cleanup" Oct 30 19:01:28.465491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465139 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8888996a-7141-4f3f-83b3-d87811232f6a" containerName="cleanup" Oct 30 19:01:28.465491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465149 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfdf45ce-60c7-44a1-91bf-21236823cbff" containerName="cleanup" Oct 30 19:01:28.465491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.465158 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad42ac0a-7ac9-4996-a585-a77c79873164" containerName="cleanup" Oct 30 19:01:28.469410 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.469387 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.504037 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.503805 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7329398d-3b90-4434-ad23-3429cb00ec52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.504037 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.503881 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.504332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.504083 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.504332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.504211 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.504332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.504301 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.504564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.504361 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.504564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.504441 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.585849 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.585819 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4"] Oct 30 19:01:28.605688 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.605653 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7329398d-3b90-4434-ad23-3429cb00ec52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.605874 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.605697 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.605874 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.605744 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkfcj\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-kube-api-access-fkfcj\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.605874 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.605792 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdktf\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-kube-api-access-gdktf\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.605874 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.605857 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7329398d-3b90-4434-ad23-3429cb00ec52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.606119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.605889 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.606119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.605996 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.606119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606026 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.606119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606058 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.606119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606104 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.606450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606137 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.606450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606142 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.606450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606167 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.606450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606426 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.606450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606439 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.606708 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606475 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.606708 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606527 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.606708 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.606662 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.607480 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.607460 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.607565 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.607493 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.607565 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.607528 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.607565 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.607550 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.610192 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.610150 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.622786 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.622763 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7329398d-3b90-4434-ad23-3429cb00ec52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.628401 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.628297 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.708916 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.708845 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkfcj\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-kube-api-access-fkfcj\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.708916 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.708908 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdktf\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-kube-api-access-gdktf\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.709142 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.708981 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7329398d-3b90-4434-ad23-3429cb00ec52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.709142 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709107 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.709281 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709142 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.709281 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709184 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.709281 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709224 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.709462 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709296 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.709462 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709385 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.709462 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709430 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.709704 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709470 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.709928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709899 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7329398d-3b90-4434-ad23-3429cb00ec52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.710039 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709945 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-data\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.710039 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709969 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-certs\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.710039 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.709944 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.710411 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.710048 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-credential-socket\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.710411 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.710302 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.711964 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.711942 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.712175 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.712157 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.736749 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.736727 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkfcj\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-kube-api-access-fkfcj\") pod \"maas-default-gateway-openshift-default-76597d6855-7lhc4\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.737863 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.737843 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdktf\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-kube-api-access-gdktf\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.738320 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.738299 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-token\") pod \"maas-default-gateway-openshift-default-76597d6855-dxjft\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:28.834383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.834356 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:28.989727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:28.989634 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" podUID="a0c69a88-80c5-4d22-a1b3-172e2f6c171d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:01:29.015513 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.015475 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:29.258012 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.257973 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4"] Oct 30 19:01:29.262777 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:01:29.262739 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7329398d_3b90_4434_ad23_3429cb00ec52.slice/crio-64cb1f2559dadd9915c2398f2f04f0f9995f262d323a5438bb66dc475c5b55ac WatchSource:0}: Error finding container 64cb1f2559dadd9915c2398f2f04f0f9995f262d323a5438bb66dc475c5b55ac: Status 404 returned error can't find the container with id 64cb1f2559dadd9915c2398f2f04f0f9995f262d323a5438bb66dc475c5b55ac Oct 30 19:01:29.267693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.267659 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:29.267821 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.267750 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:29.267821 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.267797 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:29.342080 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.342054 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj"] Oct 30 19:01:29.425891 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:01:29.425835 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd170b4b8_91b7_4857_bcbf_0f28ccb608bc.slice/crio-6e8befd1c6b344b64a5e81503136dd9c8839c010afb599e77d9bc9453c5d1368 WatchSource:0}: Error finding container 6e8befd1c6b344b64a5e81503136dd9c8839c010afb599e77d9bc9453c5d1368: Status 404 returned error can't find the container with id 6e8befd1c6b344b64a5e81503136dd9c8839c010afb599e77d9bc9453c5d1368 Oct 30 19:01:29.440517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.432003 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:29.440517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.432102 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:29.440517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.432145 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:01:29.459216 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.452910 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft"] Oct 30 19:01:29.837662 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.837617 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" event={"ID":"7329398d-3b90-4434-ad23-3429cb00ec52","Type":"ContainerStarted","Data":"173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367"} Oct 30 19:01:29.837869 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.837708 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" event={"ID":"7329398d-3b90-4434-ad23-3429cb00ec52","Type":"ContainerStarted","Data":"64cb1f2559dadd9915c2398f2f04f0f9995f262d323a5438bb66dc475c5b55ac"} Oct 30 19:01:29.839589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.839554 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" event={"ID":"d170b4b8-91b7-4857-bcbf-0f28ccb608bc","Type":"ContainerStarted","Data":"50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329"} Oct 30 19:01:29.839716 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.839610 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" event={"ID":"d170b4b8-91b7-4857-bcbf-0f28ccb608bc","Type":"ContainerStarted","Data":"6e8befd1c6b344b64a5e81503136dd9c8839c010afb599e77d9bc9453c5d1368"} Oct 30 19:01:29.960296 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:29.960200 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" podStartSLOduration=1.960182251 podStartE2EDuration="1.960182251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:29.947514808 +0000 UTC m=+3788.996559747" watchObservedRunningTime="2026-10-30 19:01:29.960182251 +0000 UTC m=+3789.009227165" Oct 30 19:01:30.016609 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:30.016566 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:30.018812 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:30.018779 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dxjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 19:01:30.018944 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:30.018834 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 19:01:30.124634 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:30.124138 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podStartSLOduration=2.124116167 podStartE2EDuration="2.124116167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:30.083751546 +0000 UTC m=+3789.132796460" watchObservedRunningTime="2026-10-30 19:01:30.124116167 +0000 UTC m=+3789.173161082" Oct 30 19:01:30.838359 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:30.838321 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:30.860344 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:30.860309 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7lhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 30 19:01:30.860634 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:30.860598 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 30 19:01:31.016172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:31.016138 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dxjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 19:01:31.016348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:31.016189 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 19:01:31.855293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:31.842003 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7lhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 30 19:01:31.855293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:31.842093 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 30 19:01:32.023291 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:32.017068 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dxjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 19:01:32.023291 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:32.017199 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 19:01:32.878289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:32.876726 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7lhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 30 19:01:32.878289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:32.876790 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 30 19:01:33.016441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:33.016411 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dxjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 19:01:33.016675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:33.016656 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 19:01:33.840290 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:33.837605 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7lhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 30 19:01:33.840290 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:33.837665 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 30 19:01:34.016414 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:34.016378 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dxjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 19:01:34.016834 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:34.016431 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 19:01:34.348940 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:34.348895 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" podUID="8dd4156c-152e-4192-9333-1f687507ee64" containerName="payload-pre-processing" containerID="cri-o://59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7" gracePeriod=25 Oct 30 19:01:34.852288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:34.851709 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7lhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 30 19:01:34.852288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:34.851781 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 30 19:01:35.016638 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:35.016602 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dxjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 19:01:35.017018 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:35.016653 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 19:01:35.845288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:35.837569 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7lhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 30 19:01:35.845288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:35.837629 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 30 19:01:36.016409 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:36.016371 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dxjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 19:01:36.016584 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:36.016435 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 19:01:36.834666 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:36.834626 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-7lhc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body= Oct 30 19:01:36.835080 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:36.834701 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" Oct 30 19:01:37.016215 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.016184 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dxjft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body= Oct 30 19:01:37.016418 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.016248 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" Oct 30 19:01:37.298662 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.298629 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 19:01:37.439784 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.439691 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-kube-api-access-8gwg7\" (UniqueName: \"kubernetes.io/projected/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-kube-api-access-8gwg7\") pod \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\" (UID: \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\") " Oct 30 19:01:37.439956 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.439809 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-plugins-config-volume\") pod \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\" (UID: \"a0c69a88-80c5-4d22-a1b3-172e2f6c171d\") " Oct 30 19:01:37.440213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.440178 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-plugins-config-volume" pod "a0c69a88-80c5-4d22-a1b3-172e2f6c171d" (UID: "a0c69a88-80c5-4d22-a1b3-172e2f6c171d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:37.440471 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.440449 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:37.443213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.443185 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-kube-api-access-8gwg7" pod "a0c69a88-80c5-4d22-a1b3-172e2f6c171d" (UID: "a0c69a88-80c5-4d22-a1b3-172e2f6c171d"). InnerVolumeSpecName "kube-api-access-8gwg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:37.542738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.542682 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gwg7\" (UniqueName: \"kubernetes.io/projected/a0c69a88-80c5-4d22-a1b3-172e2f6c171d-kube-api-access-8gwg7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:37.606692 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.606645 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" podUID="8dd4156c-152e-4192-9333-1f687507ee64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:01:37.954886 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.954849 4862 generic.go:356] "Generic (PLEG): container finished" podID="a0c69a88-80c5-4d22-a1b3-172e2f6c171d" containerID="66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb" exitCode=0 Oct 30 19:01:37.955564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.955536 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" event={"ID":"a0c69a88-80c5-4d22-a1b3-172e2f6c171d","Type":"ContainerDied","Data":"66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb"} Oct 30 19:01:37.955771 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.955738 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" event={"ID":"a0c69a88-80c5-4d22-a1b3-172e2f6c171d","Type":"ContainerDied","Data":"b110b5c9570e8886d81ea96f74da905f06e4ee5b90ca80dfd72495e18e863bf3"} Oct 30 19:01:37.955917 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.955900 4862 scope.go:122] "RemoveContainer" containerID="66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb" Oct 30 19:01:37.956203 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.956183 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k" Oct 30 19:01:37.978149 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.978123 4862 scope.go:122] "RemoveContainer" containerID="66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb" Oct 30 19:01:37.978775 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:37.978603 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb\": container with ID starting with 66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb not found: ID does not exist" containerID="66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb" Oct 30 19:01:37.978775 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:37.978642 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb"} err="failed to get container status \"66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb\": rpc error: code = NotFound desc = could not find container \"66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb\": container with ID starting with 66c412e5060f1ff1f870fb706272d36b61407175608d25607a7432315b4c06cb not found: ID does not exist" Oct 30 19:01:38.055777 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:38.055727 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k"] Oct 30 19:01:38.063996 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:38.063953 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8862aa-7d764b8787-8ln4k"] Oct 30 19:01:38.310019 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:38.309980 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:38.311066 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:38.311029 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:38.311839 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:38.311814 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:01:38.677468 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:38.677385 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:38.678024 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:38.678001 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:38.678961 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:38.678942 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:01:39.064011 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:39.063980 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c69a88-80c5-4d22-a1b3-172e2f6c171d" path="/var/lib/kubelet/pods/a0c69a88-80c5-4d22-a1b3-172e2f6c171d/volumes" Oct 30 19:01:47.606094 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:47.606056 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" podUID="8dd4156c-152e-4192-9333-1f687507ee64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:01:49.324994 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:49.324961 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:01:49.885828 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:49.885792 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:01:54.565417 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:54.565385 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 19:01:54.660984 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:54.660916 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd4156c-152e-4192-9333-1f687507ee64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4156c-152e-4192-9333-1f687507ee64-plugins-config-volume\") pod \"8dd4156c-152e-4192-9333-1f687507ee64\" (UID: \"8dd4156c-152e-4192-9333-1f687507ee64\") " Oct 30 19:01:54.661154 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:54.661047 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd4156c-152e-4192-9333-1f687507ee64-kube-api-access-n4wfm\" (UniqueName: \"kubernetes.io/projected/8dd4156c-152e-4192-9333-1f687507ee64-kube-api-access-n4wfm\") pod \"8dd4156c-152e-4192-9333-1f687507ee64\" (UID: \"8dd4156c-152e-4192-9333-1f687507ee64\") " Oct 30 19:01:54.661393 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:54.661367 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd4156c-152e-4192-9333-1f687507ee64-plugins-config-volume" pod "8dd4156c-152e-4192-9333-1f687507ee64" (UID: "8dd4156c-152e-4192-9333-1f687507ee64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:54.661788 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:54.661758 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd4156c-152e-4192-9333-1f687507ee64-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:54.663510 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:54.663486 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd4156c-152e-4192-9333-1f687507ee64-kube-api-access-n4wfm" pod "8dd4156c-152e-4192-9333-1f687507ee64" (UID: "8dd4156c-152e-4192-9333-1f687507ee64"). InnerVolumeSpecName "kube-api-access-n4wfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:54.763008 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:54.762979 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4wfm\" (UniqueName: \"kubernetes.io/projected/8dd4156c-152e-4192-9333-1f687507ee64-kube-api-access-n4wfm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:01:55.080053 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:55.080018 4862 generic.go:356] "Generic (PLEG): container finished" podID="8dd4156c-152e-4192-9333-1f687507ee64" containerID="59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7" exitCode=0 Oct 30 19:01:55.080197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:55.080062 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" Oct 30 19:01:55.080197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:55.080102 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" event={"ID":"8dd4156c-152e-4192-9333-1f687507ee64","Type":"ContainerDied","Data":"59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7"} Oct 30 19:01:55.080197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:55.080150 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj" event={"ID":"8dd4156c-152e-4192-9333-1f687507ee64","Type":"ContainerDied","Data":"087b8257e67f403ed0d31667371a07937007017af15f357b7f7f1680ee5dc68d"} Oct 30 19:01:55.080197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:55.080172 4862 scope.go:122] "RemoveContainer" containerID="59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7" Oct 30 19:01:55.091033 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:55.091015 4862 scope.go:122] "RemoveContainer" containerID="59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7" Oct 30 19:01:55.091316 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:01:55.091294 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7\": container with ID starting with 59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7 not found: ID does not exist" containerID="59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7" Oct 30 19:01:55.091389 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:55.091327 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7"} err="failed to get container status \"59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7\": rpc error: code = NotFound desc = could not find container \"59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7\": container with ID starting with 59efd6c3d1b76803d2cae61fc9f2aa99c0d3537dbf1670482e612c0be714e9f7 not found: ID does not exist" Oct 30 19:01:55.139089 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:55.139062 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj"] Oct 30 19:01:55.152480 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:55.152459 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-551d0b-78b87bfb99-xtrvj"] Oct 30 19:01:57.063514 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:01:57.063480 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd4156c-152e-4192-9333-1f687507ee64" path="/var/lib/kubelet/pods/8dd4156c-152e-4192-9333-1f687507ee64/volumes" Oct 30 19:02:00.513626 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:00.513506 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv"] Oct 30 19:02:00.584360 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:00.584328 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml"] Oct 30 19:02:05.514986 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:05.514907 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" podUID="0cb6ab44-54dc-4284-8e09-5d4a5d756d23" containerName="payload-processing" containerID="cri-o://74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8" gracePeriod=25 Oct 30 19:02:05.585053 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:05.585014 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" podUID="a3487105-c424-45bd-9913-3f90091b0e70" containerName="payload-pre-processing" containerID="cri-o://b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e" gracePeriod=25 Oct 30 19:02:05.800756 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:05.800732 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 19:02:05.931675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:05.931634 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-kube-api-access-q5bt2\" (UniqueName: \"kubernetes.io/projected/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-kube-api-access-q5bt2\") pod \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\" (UID: \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\") " Oct 30 19:02:05.931872 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:05.931785 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-plugins-config-volume\") pod \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\" (UID: \"0cb6ab44-54dc-4284-8e09-5d4a5d756d23\") " Oct 30 19:02:05.932217 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:05.932194 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-plugins-config-volume" pod "0cb6ab44-54dc-4284-8e09-5d4a5d756d23" (UID: "0cb6ab44-54dc-4284-8e09-5d4a5d756d23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:05.932431 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:05.932417 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:05.933876 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:05.933860 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-kube-api-access-q5bt2" pod "0cb6ab44-54dc-4284-8e09-5d4a5d756d23" (UID: "0cb6ab44-54dc-4284-8e09-5d4a5d756d23"). InnerVolumeSpecName "kube-api-access-q5bt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:06.033338 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.033249 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5bt2\" (UniqueName: \"kubernetes.io/projected/0cb6ab44-54dc-4284-8e09-5d4a5d756d23-kube-api-access-q5bt2\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:06.182090 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.182063 4862 generic.go:356] "Generic (PLEG): container finished" podID="0cb6ab44-54dc-4284-8e09-5d4a5d756d23" containerID="74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8" exitCode=0 Oct 30 19:02:06.182228 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.182110 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" Oct 30 19:02:06.182228 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.182149 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" event={"ID":"0cb6ab44-54dc-4284-8e09-5d4a5d756d23","Type":"ContainerDied","Data":"74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8"} Oct 30 19:02:06.182228 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.182202 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv" event={"ID":"0cb6ab44-54dc-4284-8e09-5d4a5d756d23","Type":"ContainerDied","Data":"29a70e8f552c3c001f91f9014cef7bbe35021bea6f13a3bcf5152da0fe0b2e11"} Oct 30 19:02:06.182407 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.182230 4862 scope.go:122] "RemoveContainer" containerID="74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8" Oct 30 19:02:06.193467 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.193452 4862 scope.go:122] "RemoveContainer" containerID="74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8" Oct 30 19:02:06.193776 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:02:06.193758 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8\": container with ID starting with 74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8 not found: ID does not exist" containerID="74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8" Oct 30 19:02:06.193847 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.193784 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8"} err="failed to get container status \"74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8\": rpc error: code = NotFound desc = could not find container \"74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8\": container with ID starting with 74bf2c97bc9a585648d3c3a5459ed62a05bfab3b6496a66c3696c06014024fd8 not found: ID does not exist" Oct 30 19:02:06.237160 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.236239 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv"] Oct 30 19:02:06.242743 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:06.242723 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f37ca7-85fcd85f88-xc8cv"] Oct 30 19:02:07.075853 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:07.074147 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb6ab44-54dc-4284-8e09-5d4a5d756d23" path="/var/lib/kubelet/pods/0cb6ab44-54dc-4284-8e09-5d4a5d756d23/volumes" Oct 30 19:02:12.524392 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:12.524357 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" podUID="a3487105-c424-45bd-9913-3f90091b0e70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:02:19.872886 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:02:19.872848 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:02:22.525087 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:22.525044 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" podUID="a3487105-c424-45bd-9913-3f90091b0e70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:02:25.872639 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:25.872605 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 19:02:25.951068 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:25.951027 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3487105-c424-45bd-9913-3f90091b0e70-kube-api-access-7vpt5\" (UniqueName: \"kubernetes.io/projected/a3487105-c424-45bd-9913-3f90091b0e70-kube-api-access-7vpt5\") pod \"a3487105-c424-45bd-9913-3f90091b0e70\" (UID: \"a3487105-c424-45bd-9913-3f90091b0e70\") " Oct 30 19:02:25.951255 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:25.951169 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3487105-c424-45bd-9913-3f90091b0e70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3487105-c424-45bd-9913-3f90091b0e70-plugins-config-volume\") pod \"a3487105-c424-45bd-9913-3f90091b0e70\" (UID: \"a3487105-c424-45bd-9913-3f90091b0e70\") " Oct 30 19:02:25.951670 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:25.951639 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3487105-c424-45bd-9913-3f90091b0e70-plugins-config-volume" pod "a3487105-c424-45bd-9913-3f90091b0e70" (UID: "a3487105-c424-45bd-9913-3f90091b0e70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:25.951986 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:25.951962 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3487105-c424-45bd-9913-3f90091b0e70-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:25.953485 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:25.953460 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3487105-c424-45bd-9913-3f90091b0e70-kube-api-access-7vpt5" pod "a3487105-c424-45bd-9913-3f90091b0e70" (UID: "a3487105-c424-45bd-9913-3f90091b0e70"). InnerVolumeSpecName "kube-api-access-7vpt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:26.053506 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.053476 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vpt5\" (UniqueName: \"kubernetes.io/projected/a3487105-c424-45bd-9913-3f90091b0e70-kube-api-access-7vpt5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:26.368355 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.368316 4862 generic.go:356] "Generic (PLEG): container finished" podID="a3487105-c424-45bd-9913-3f90091b0e70" containerID="b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e" exitCode=0 Oct 30 19:02:26.368524 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.368371 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" Oct 30 19:02:26.368524 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.368399 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" event={"ID":"a3487105-c424-45bd-9913-3f90091b0e70","Type":"ContainerDied","Data":"b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e"} Oct 30 19:02:26.368524 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.368461 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml" event={"ID":"a3487105-c424-45bd-9913-3f90091b0e70","Type":"ContainerDied","Data":"0aeca0c59cc1baa2a43c0f702a35a1ea160a5a41bbbfefcaad5df0d97d425c2b"} Oct 30 19:02:26.368524 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.368483 4862 scope.go:122] "RemoveContainer" containerID="b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e" Oct 30 19:02:26.379688 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.379660 4862 scope.go:122] "RemoveContainer" containerID="b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e" Oct 30 19:02:26.379992 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:02:26.379944 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e\": container with ID starting with b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e not found: ID does not exist" containerID="b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e" Oct 30 19:02:26.379992 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.379973 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e"} err="failed to get container status \"b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e\": rpc error: code = NotFound desc = could not find container \"b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e\": container with ID starting with b1bc769dfee68fc17d74145ca07a9ffc7a33cf884411e7aec76228599ee5ce5e not found: ID does not exist" Oct 30 19:02:26.418952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.418924 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml"] Oct 30 19:02:26.431309 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:26.431284 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f37ca7-859dd99c87-d64ml"] Oct 30 19:02:27.063211 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:27.063181 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3487105-c424-45bd-9913-3f90091b0e70" path="/var/lib/kubelet/pods/a3487105-c424-45bd-9913-3f90091b0e70/volumes" Oct 30 19:02:38.206390 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:38.206349 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9"] Oct 30 19:02:38.206889 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:38.206610 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" containerID="cri-o://44b17779e9df9ec48e1d486c82e3436f1b8a05105ff53cb7935ae16275dfaebe" gracePeriod=30 Oct 30 19:02:38.215942 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:38.215906 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7"] Oct 30 19:02:38.216499 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:38.216428 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" containerID="cri-o://84aef472547aa68285d3b6fece3a41aa2634e3fb5ebfd8eeea7c5f90f357d963" gracePeriod=30 Oct 30 19:02:39.559801 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:39.559765 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:02:40.041442 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:40.041339 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26"] Oct 30 19:02:40.047495 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:40.047463 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a70db8-h4b26"] Oct 30 19:02:41.065213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:41.065173 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff36e59a-d24d-413c-80aa-d937ec1279ac" path="/var/lib/kubelet/pods/ff36e59a-d24d-413c-80aa-d937ec1279ac/volumes" Oct 30 19:02:43.713161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.712855 4862 generic.go:356] "Generic (PLEG): container finished" podID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerID="84aef472547aa68285d3b6fece3a41aa2634e3fb5ebfd8eeea7c5f90f357d963" exitCode=0 Oct 30 19:02:43.713161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.713021 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" event={"ID":"6f8a371c-43aa-4fca-bf65-c22c553cbb8c","Type":"ContainerDied","Data":"84aef472547aa68285d3b6fece3a41aa2634e3fb5ebfd8eeea7c5f90f357d963"} Oct 30 19:02:43.713161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.713071 4862 scope.go:122] "RemoveContainer" containerID="3e56ebb0bd4644d0096aefd6d2a89031015022e90b0bd86b97f2a17deceb0510" Oct 30 19:02:43.716347 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.716325 4862 generic.go:356] "Generic (PLEG): container finished" podID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerID="44b17779e9df9ec48e1d486c82e3436f1b8a05105ff53cb7935ae16275dfaebe" exitCode=0 Oct 30 19:02:43.716525 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.716470 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" event={"ID":"558226ee-2091-4c41-a0e8-70591a91a3cd","Type":"ContainerDied","Data":"44b17779e9df9ec48e1d486c82e3436f1b8a05105ff53cb7935ae16275dfaebe"} Oct 30 19:02:43.814297 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.814210 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 19:02:43.909632 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.909609 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 19:02:43.916993 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.916971 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558226ee-2091-4c41-a0e8-70591a91a3cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558226ee-2091-4c41-a0e8-70591a91a3cd-istiod-ca-cert\") pod \"558226ee-2091-4c41-a0e8-70591a91a3cd\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " Oct 30 19:02:43.917152 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917020 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-certs\") pod \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " Oct 30 19:02:43.917152 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917074 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-envoy\") pod \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " Oct 30 19:02:43.917152 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917144 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-kube-api-access-97g98\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-kube-api-access-97g98\") pod \"558226ee-2091-4c41-a0e8-70591a91a3cd\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " Oct 30 19:02:43.917354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917210 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-credential-socket\") pod \"558226ee-2091-4c41-a0e8-70591a91a3cd\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " Oct 30 19:02:43.917354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917307 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-certs\") pod \"558226ee-2091-4c41-a0e8-70591a91a3cd\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " Oct 30 19:02:43.917462 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917368 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istiod-ca-cert\") pod \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " Oct 30 19:02:43.917462 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917444 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-socket\") pod \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " Oct 30 19:02:43.917583 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917491 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-socket\") pod \"558226ee-2091-4c41-a0e8-70591a91a3cd\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " Oct 30 19:02:43.917583 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917568 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-credential-socket\") pod \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " Oct 30 19:02:43.917691 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917601 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-data\") pod \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " Oct 30 19:02:43.917750 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917720 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-token\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-token\") pod \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917863 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-podinfo\") pod \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917916 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-token\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-token\") pod \"558226ee-2091-4c41-a0e8-70591a91a3cd\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.917975 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-podinfo\") pod \"558226ee-2091-4c41-a0e8-70591a91a3cd\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918025 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-data\") pod \"558226ee-2091-4c41-a0e8-70591a91a3cd\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918046 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-certs" pod "558226ee-2091-4c41-a0e8-70591a91a3cd" (UID: "558226ee-2091-4c41-a0e8-70591a91a3cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918070 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istiod-ca-cert" pod "6f8a371c-43aa-4fca-bf65-c22c553cbb8c" (UID: "6f8a371c-43aa-4fca-bf65-c22c553cbb8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918122 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-kube-api-access-zkcd9\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-kube-api-access-zkcd9\") pod \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\" (UID: \"6f8a371c-43aa-4fca-bf65-c22c553cbb8c\") " Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918162 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-envoy\") pod \"558226ee-2091-4c41-a0e8-70591a91a3cd\" (UID: \"558226ee-2091-4c41-a0e8-70591a91a3cd\") " Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918311 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-data" pod "558226ee-2091-4c41-a0e8-70591a91a3cd" (UID: "558226ee-2091-4c41-a0e8-70591a91a3cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918379 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-credential-socket" pod "558226ee-2091-4c41-a0e8-70591a91a3cd" (UID: "558226ee-2091-4c41-a0e8-70591a91a3cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918592 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-credential-socket" pod "6f8a371c-43aa-4fca-bf65-c22c553cbb8c" (UID: "6f8a371c-43aa-4fca-bf65-c22c553cbb8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918770 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918780 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-data" pod "6f8a371c-43aa-4fca-bf65-c22c553cbb8c" (UID: "6f8a371c-43aa-4fca-bf65-c22c553cbb8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918786 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918822 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918838 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918855 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:43.919098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.918905 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-socket" pod "6f8a371c-43aa-4fca-bf65-c22c553cbb8c" (UID: "6f8a371c-43aa-4fca-bf65-c22c553cbb8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.920360 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.919404 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-socket" pod "558226ee-2091-4c41-a0e8-70591a91a3cd" (UID: "558226ee-2091-4c41-a0e8-70591a91a3cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.921245 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.920618 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558226ee-2091-4c41-a0e8-70591a91a3cd-istiod-ca-cert" pod "558226ee-2091-4c41-a0e8-70591a91a3cd" (UID: "558226ee-2091-4c41-a0e8-70591a91a3cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:43.921245 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.920895 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-certs" pod "6f8a371c-43aa-4fca-bf65-c22c553cbb8c" (UID: "6f8a371c-43aa-4fca-bf65-c22c553cbb8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.921560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.921537 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-envoy" pod "558226ee-2091-4c41-a0e8-70591a91a3cd" (UID: "558226ee-2091-4c41-a0e8-70591a91a3cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.921900 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.921875 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-envoy" pod "6f8a371c-43aa-4fca-bf65-c22c553cbb8c" (UID: "6f8a371c-43aa-4fca-bf65-c22c553cbb8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:02:43.921987 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.921961 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-token" pod "6f8a371c-43aa-4fca-bf65-c22c553cbb8c" (UID: "6f8a371c-43aa-4fca-bf65-c22c553cbb8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:43.922294 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.922252 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-kube-api-access-zkcd9" pod "6f8a371c-43aa-4fca-bf65-c22c553cbb8c" (UID: "6f8a371c-43aa-4fca-bf65-c22c553cbb8c"). InnerVolumeSpecName "kube-api-access-zkcd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:43.923638 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.923606 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-podinfo" pod "558226ee-2091-4c41-a0e8-70591a91a3cd" (UID: "558226ee-2091-4c41-a0e8-70591a91a3cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:02:43.923952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.923928 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-token" pod "558226ee-2091-4c41-a0e8-70591a91a3cd" (UID: "558226ee-2091-4c41-a0e8-70591a91a3cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:43.924025 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.923950 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-podinfo" pod "6f8a371c-43aa-4fca-bf65-c22c553cbb8c" (UID: "6f8a371c-43aa-4fca-bf65-c22c553cbb8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:02:43.925115 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:43.925081 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-kube-api-access-97g98" pod "558226ee-2091-4c41-a0e8-70591a91a3cd" (UID: "558226ee-2091-4c41-a0e8-70591a91a3cd"). InnerVolumeSpecName "kube-api-access-97g98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:44.020280 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020169 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020280 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020213 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020280 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020228 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97g98\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-kube-api-access-97g98\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020280 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020245 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020280 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020278 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020293 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020306 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020318 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020331 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020344 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020356 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkcd9\" (UniqueName: \"kubernetes.io/projected/6f8a371c-43aa-4fca-bf65-c22c553cbb8c-kube-api-access-zkcd9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020369 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/558226ee-2091-4c41-a0e8-70591a91a3cd-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.020762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.020382 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/558226ee-2091-4c41-a0e8-70591a91a3cd-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:44.725288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.725227 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" Oct 30 19:02:44.725288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.725285 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7" event={"ID":"6f8a371c-43aa-4fca-bf65-c22c553cbb8c","Type":"ContainerDied","Data":"81e2b078c7aa6ce8b035e4c1724dc52bd60002dc60f6296766d7032aaeca17af"} Oct 30 19:02:44.725858 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.725338 4862 scope.go:122] "RemoveContainer" containerID="84aef472547aa68285d3b6fece3a41aa2634e3fb5ebfd8eeea7c5f90f357d963" Oct 30 19:02:44.727088 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.727064 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" event={"ID":"558226ee-2091-4c41-a0e8-70591a91a3cd","Type":"ContainerDied","Data":"e91950b6fe9af3482aaec300498cc2fb5bd4a8fad8a246073f553591845dcbba"} Oct 30 19:02:44.727234 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.727164 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9" Oct 30 19:02:44.739381 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.739360 4862 scope.go:122] "RemoveContainer" containerID="44b17779e9df9ec48e1d486c82e3436f1b8a05105ff53cb7935ae16275dfaebe" Oct 30 19:02:44.789637 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.789613 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9"] Oct 30 19:02:44.798767 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.798746 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-cljg9"] Oct 30 19:02:44.824712 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.824685 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7"] Oct 30 19:02:44.833381 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:44.833360 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-551d0b-openshift-default-57c5b6c45f-9kzp7"] Oct 30 19:02:45.063507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:45.063478 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" path="/var/lib/kubelet/pods/558226ee-2091-4c41-a0e8-70591a91a3cd/volumes" Oct 30 19:02:45.063972 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:45.063958 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" path="/var/lib/kubelet/pods/6f8a371c-43aa-4fca-bf65-c22c553cbb8c/volumes" Oct 30 19:02:49.869539 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:02:49.869496 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:02:52.028944 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:52.028909 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk"] Oct 30 19:02:52.038942 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:52.038917 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862387e2-wr2bk"] Oct 30 19:02:53.063682 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:53.063647 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f6d1a9-c05c-4f9a-9057-8c6d94609e8d" path="/var/lib/kubelet/pods/40f6d1a9-c05c-4f9a-9057-8c6d94609e8d/volumes" Oct 30 19:02:57.367726 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.367689 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2"] Oct 30 19:02:57.368626 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.367948 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" podUID="a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" containerName="maas-api" containerID="cri-o://fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b" gracePeriod=30 Oct 30 19:02:57.841476 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.841443 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 19:02:57.894287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.885515 4862 generic.go:356] "Generic (PLEG): container finished" podID="a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" containerID="fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b" exitCode=0 Oct 30 19:02:57.894287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.885720 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" event={"ID":"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532","Type":"ContainerDied","Data":"fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b"} Oct 30 19:02:57.894287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.885759 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" event={"ID":"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532","Type":"ContainerDied","Data":"5d9ef88c9004617b30787e9c84d67ebf13d39a91d1b7ee12ead211d489772cf8"} Oct 30 19:02:57.894287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.885779 4862 scope.go:122] "RemoveContainer" containerID="fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b" Oct 30 19:02:57.894287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.885905 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2" Oct 30 19:02:57.931288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.926178 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-kube-api-access-7dmgh\" (UniqueName: \"kubernetes.io/projected/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-kube-api-access-7dmgh\") pod \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " Oct 30 19:02:57.931288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.926234 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-maas-api-tls\") pod \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " Oct 30 19:02:57.931288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.926603 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-openshift-service-ca\") pod \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " Oct 30 19:02:57.931288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.926688 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-metrics-certs\") pod \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\" (UID: \"a37c63f2-e0ec-4f50-bc8e-98f9a0e19532\") " Oct 30 19:02:57.931288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.930150 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-openshift-service-ca" pod "a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" (UID: "a37c63f2-e0ec-4f50-bc8e-98f9a0e19532"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:02:57.931288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.931046 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-maas-api-tls" pod "a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" (UID: "a37c63f2-e0ec-4f50-bc8e-98f9a0e19532"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:02:57.949721 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.948752 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-metrics-certs" pod "a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" (UID: "a37c63f2-e0ec-4f50-bc8e-98f9a0e19532"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:02:57.954986 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.954954 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-kube-api-access-7dmgh" pod "a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" (UID: "a37c63f2-e0ec-4f50-bc8e-98f9a0e19532"). InnerVolumeSpecName "kube-api-access-7dmgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:57.980997 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.980957 4862 scope.go:122] "RemoveContainer" containerID="fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b" Oct 30 19:02:57.981655 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:02:57.981620 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b\": container with ID starting with fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b not found: ID does not exist" containerID="fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b" Oct 30 19:02:57.981790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:57.981667 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b"} err="failed to get container status \"fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b\": rpc error: code = NotFound desc = could not find container \"fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b\": container with ID starting with fa386c1eb41c7ea6bc0e9abffc4f185b1a2c91a7c17d1f933511667fe79fa21b not found: ID does not exist" Oct 30 19:02:58.029207 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:58.029126 4862 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-openshift-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:58.029207 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:58.029155 4862 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-metrics-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:58.029207 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:58.029170 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dmgh\" (UniqueName: \"kubernetes.io/projected/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-kube-api-access-7dmgh\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:58.029207 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:58.029182 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532-maas-api-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:02:58.230896 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:58.230864 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2"] Oct 30 19:02:58.240611 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:58.240585 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f37ca7-7cdfcd9c65-wbmk2"] Oct 30 19:02:59.063500 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:02:59.063464 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" path="/var/lib/kubelet/pods/a37c63f2-e0ec-4f50-bc8e-98f9a0e19532/volumes" Oct 30 19:03:05.102464 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:05.102427 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml"] Oct 30 19:03:05.103663 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:05.103566 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerName="istio-proxy" containerID="cri-o://a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6" gracePeriod=30 Oct 30 19:03:10.392826 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.392793 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 19:03:10.507462 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507428 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-data\") pod \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " Oct 30 19:03:10.507656 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507521 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istiod-ca-cert\") pod \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " Oct 30 19:03:10.507656 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507633 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-kube-api-access-fmrjx\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-kube-api-access-fmrjx\") pod \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " Oct 30 19:03:10.507797 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507702 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-envoy\") pod \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " Oct 30 19:03:10.507797 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507737 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-socket\") pod \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " Oct 30 19:03:10.507797 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507731 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-data" pod "90e18bf6-97c4-46a5-83a6-1f7ec21655f1" (UID: "90e18bf6-97c4-46a5-83a6-1f7ec21655f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.507964 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507879 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-podinfo\") pod \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " Oct 30 19:03:10.507964 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507878 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istiod-ca-cert" pod "90e18bf6-97c4-46a5-83a6-1f7ec21655f1" (UID: "90e18bf6-97c4-46a5-83a6-1f7ec21655f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:03:10.507964 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507949 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-credential-socket\") pod \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " Oct 30 19:03:10.508132 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507978 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-certs\") pod \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " Oct 30 19:03:10.508132 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.507999 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-socket" pod "90e18bf6-97c4-46a5-83a6-1f7ec21655f1" (UID: "90e18bf6-97c4-46a5-83a6-1f7ec21655f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.508132 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.508087 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-token\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-token\") pod \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\" (UID: \"90e18bf6-97c4-46a5-83a6-1f7ec21655f1\") " Oct 30 19:03:10.508308 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.508237 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-credential-socket" pod "90e18bf6-97c4-46a5-83a6-1f7ec21655f1" (UID: "90e18bf6-97c4-46a5-83a6-1f7ec21655f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.508308 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.508250 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-certs" pod "90e18bf6-97c4-46a5-83a6-1f7ec21655f1" (UID: "90e18bf6-97c4-46a5-83a6-1f7ec21655f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.508748 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.508727 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.508857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.508752 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.508857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.508765 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.508857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.508777 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.508857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.508791 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.510332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.510304 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-podinfo" pod "90e18bf6-97c4-46a5-83a6-1f7ec21655f1" (UID: "90e18bf6-97c4-46a5-83a6-1f7ec21655f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:03:10.510638 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.510615 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-envoy" pod "90e18bf6-97c4-46a5-83a6-1f7ec21655f1" (UID: "90e18bf6-97c4-46a5-83a6-1f7ec21655f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:03:10.510753 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.510655 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-kube-api-access-fmrjx" pod "90e18bf6-97c4-46a5-83a6-1f7ec21655f1" (UID: "90e18bf6-97c4-46a5-83a6-1f7ec21655f1"). InnerVolumeSpecName "kube-api-access-fmrjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:10.510987 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.510963 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-token" pod "90e18bf6-97c4-46a5-83a6-1f7ec21655f1" (UID: "90e18bf6-97c4-46a5-83a6-1f7ec21655f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:03:10.610103 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.610071 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.610103 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.610100 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.610324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.610114 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmrjx\" (UniqueName: \"kubernetes.io/projected/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-kube-api-access-fmrjx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:03:10.610324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:10.610127 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e18bf6-97c4-46a5-83a6-1f7ec21655f1-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:03:11.044735 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:11.044703 4862 generic.go:356] "Generic (PLEG): container finished" podID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerID="a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6" exitCode=0 Oct 30 19:03:11.044965 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:11.044755 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" Oct 30 19:03:11.044965 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:11.044805 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" event={"ID":"90e18bf6-97c4-46a5-83a6-1f7ec21655f1","Type":"ContainerDied","Data":"a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6"} Oct 30 19:03:11.044965 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:11.044860 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml" event={"ID":"90e18bf6-97c4-46a5-83a6-1f7ec21655f1","Type":"ContainerDied","Data":"3e55913dda9d47a7627a41389d4d9a56a1f7f453cf7ee436f3b9e79b745ef08c"} Oct 30 19:03:11.044965 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:11.044887 4862 scope.go:122] "RemoveContainer" containerID="a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6" Oct 30 19:03:11.057532 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:11.057508 4862 scope.go:122] "RemoveContainer" containerID="a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6" Oct 30 19:03:11.057801 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:03:11.057782 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6\": container with ID starting with a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6 not found: ID does not exist" containerID="a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6" Oct 30 19:03:11.057874 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:11.057809 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6"} err="failed to get container status \"a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6\": rpc error: code = NotFound desc = could not find container \"a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6\": container with ID starting with a9c394d83386a8ccb6914e1e9b7e8ac23a4e242941fc42496100fc5b690b45a6 not found: ID does not exist" Oct 30 19:03:11.103354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:11.103327 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml"] Oct 30 19:03:11.113770 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:11.113747 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f37ca7-openshift-default-8f7fb5559-889ml"] Oct 30 19:03:13.063894 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:13.063863 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" path="/var/lib/kubelet/pods/90e18bf6-97c4-46a5-83a6-1f7ec21655f1/volumes" Oct 30 19:03:19.872717 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:03:19.872677 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:03:21.960994 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:21.960963 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 19:03:21.966328 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:21.966302 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 19:03:31.747575 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:03:31.747483 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:03:49.869950 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:03:49.869915 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:04:10.537159 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537121 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l"] Oct 30 19:04:10.537812 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537796 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" Oct 30 19:04:10.537859 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537815 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" Oct 30 19:04:10.537859 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537832 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" Oct 30 19:04:10.537859 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537837 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" Oct 30 19:04:10.537859 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537853 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8dd4156c-152e-4192-9333-1f687507ee64" containerName="payload-pre-processing" Oct 30 19:04:10.537859 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537858 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd4156c-152e-4192-9333-1f687507ee64" containerName="payload-pre-processing" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537867 4862 cpu_manager.go:414] "containerMap: removing container" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537871 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537888 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a0c69a88-80c5-4d22-a1b3-172e2f6c171d" containerName="payload-pre-processing" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537893 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c69a88-80c5-4d22-a1b3-172e2f6c171d" containerName="payload-pre-processing" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537903 4862 cpu_manager.go:414] "containerMap: removing container" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerName="istio-proxy" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537907 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerName="istio-proxy" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537916 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a3487105-c424-45bd-9913-3f90091b0e70" containerName="payload-pre-processing" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537920 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3487105-c424-45bd-9913-3f90091b0e70" containerName="payload-pre-processing" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537926 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" containerName="maas-api" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537930 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" containerName="maas-api" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537946 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0cb6ab44-54dc-4284-8e09-5d4a5d756d23" containerName="payload-processing" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.537951 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb6ab44-54dc-4284-8e09-5d4a5d756d23" containerName="payload-processing" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.538059 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.538067 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f8a371c-43aa-4fca-bf65-c22c553cbb8c" containerName="istio-proxy" Oct 30 19:04:10.538077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.538077 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd4156c-152e-4192-9333-1f687507ee64" containerName="payload-pre-processing" Oct 30 19:04:10.538586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.538086 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="558226ee-2091-4c41-a0e8-70591a91a3cd" containerName="istio-proxy" Oct 30 19:04:10.538586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.538097 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c69a88-80c5-4d22-a1b3-172e2f6c171d" containerName="payload-pre-processing" Oct 30 19:04:10.538586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.538106 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e18bf6-97c4-46a5-83a6-1f7ec21655f1" containerName="istio-proxy" Oct 30 19:04:10.538586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.538112 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3487105-c424-45bd-9913-3f90091b0e70" containerName="payload-pre-processing" Oct 30 19:04:10.538586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.538118 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37c63f2-e0ec-4f50-bc8e-98f9a0e19532" containerName="maas-api" Oct 30 19:04:10.538586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.538126 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb6ab44-54dc-4284-8e09-5d4a5d756d23" containerName="payload-processing" Oct 30 19:04:10.541811 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.541793 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.565620 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.565549 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:04:10.565620 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.565595 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:10.571874 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.571829 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l"] Oct 30 19:04:10.582933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.582909 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9411391-7gv2l\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.583077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.582951 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfd69\" (UniqueName: \"kubernetes.io/projected/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-kube-api-access-kfd69\") pod \"maas-api-revoke-sub-f9411391-7gv2l\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.583121 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.583086 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9411391-7gv2l\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.684370 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.684335 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9411391-7gv2l\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.684557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.684389 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfd69\" (UniqueName: \"kubernetes.io/projected/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-kube-api-access-kfd69\") pod \"maas-api-revoke-sub-f9411391-7gv2l\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.684557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.684458 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9411391-7gv2l\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.685066 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.685043 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9411391-7gv2l\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.687039 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.687014 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9411391-7gv2l\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.704450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.704431 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfd69\" (UniqueName: \"kubernetes.io/projected/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-kube-api-access-kfd69\") pod \"maas-api-revoke-sub-f9411391-7gv2l\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:10.855970 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:10.855931 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:11.186896 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:11.186597 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l"] Oct 30 19:04:11.188302 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:04:11.188256 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2a0513_e682_4a1e_aa20_c6f3e310ffab.slice/crio-7d9b331835de78f49138f91b463690c8462f822ba3dd9e3cade1130793792038 WatchSource:0}: Error finding container 7d9b331835de78f49138f91b463690c8462f822ba3dd9e3cade1130793792038: Status 404 returned error can't find the container with id 7d9b331835de78f49138f91b463690c8462f822ba3dd9e3cade1130793792038 Oct 30 19:04:11.487593 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:11.487561 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" event={"ID":"5f2a0513-e682-4a1e-aa20-c6f3e310ffab","Type":"ContainerStarted","Data":"7d9b331835de78f49138f91b463690c8462f822ba3dd9e3cade1130793792038"} Oct 30 19:04:12.498011 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:12.497917 4862 generic.go:356] "Generic (PLEG): container finished" podID="5f2a0513-e682-4a1e-aa20-c6f3e310ffab" containerID="3118f0d85ce6c9e640cb58cb90c3aadf7740b54535fe1ec7e7ba1b818998d6fb" exitCode=0 Oct 30 19:04:12.498464 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:12.498008 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" event={"ID":"5f2a0513-e682-4a1e-aa20-c6f3e310ffab","Type":"ContainerDied","Data":"3118f0d85ce6c9e640cb58cb90c3aadf7740b54535fe1ec7e7ba1b818998d6fb"} Oct 30 19:04:13.690333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.690307 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:13.832158 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.832125 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-openshift-ca\") pod \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " Oct 30 19:04:13.832352 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.832318 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-kube-api-access-kfd69\" (UniqueName: \"kubernetes.io/projected/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-kube-api-access-kfd69\") pod \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " Oct 30 19:04:13.832455 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.832430 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-xks-ca\") pod \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\" (UID: \"5f2a0513-e682-4a1e-aa20-c6f3e310ffab\") " Oct 30 19:04:13.832521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.832499 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-openshift-ca" pod "5f2a0513-e682-4a1e-aa20-c6f3e310ffab" (UID: "5f2a0513-e682-4a1e-aa20-c6f3e310ffab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:13.832980 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.832960 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:13.834502 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.834476 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-kube-api-access-kfd69" pod "5f2a0513-e682-4a1e-aa20-c6f3e310ffab" (UID: "5f2a0513-e682-4a1e-aa20-c6f3e310ffab"). InnerVolumeSpecName "kube-api-access-kfd69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:13.834582 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.834545 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-xks-ca" pod "5f2a0513-e682-4a1e-aa20-c6f3e310ffab" (UID: "5f2a0513-e682-4a1e-aa20-c6f3e310ffab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:13.934589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.934560 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfd69\" (UniqueName: \"kubernetes.io/projected/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-kube-api-access-kfd69\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:13.934589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:13.934588 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2a0513-e682-4a1e-aa20-c6f3e310ffab-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:14.517083 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:14.517052 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" event={"ID":"5f2a0513-e682-4a1e-aa20-c6f3e310ffab","Type":"ContainerDied","Data":"7d9b331835de78f49138f91b463690c8462f822ba3dd9e3cade1130793792038"} Oct 30 19:04:14.517293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:14.517086 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l" Oct 30 19:04:14.517293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:14.517110 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d9b331835de78f49138f91b463690c8462f822ba3dd9e3cade1130793792038" Oct 30 19:04:19.889936 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:04:19.889900 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:04:21.406495 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:21.406442 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj"] Oct 30 19:04:23.561345 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:23.561315 4862 scope.go:122] "RemoveContainer" containerID="b2d51e6c03d9bee13790279574acff08d4baa6101a6460c506d6efbc62179c29" Oct 30 19:04:23.572354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:23.572334 4862 scope.go:122] "RemoveContainer" containerID="b7186296ea7758dc169ca0b3c6b3ca3e4569399840fc398edfefc1f689e4a3cc" Oct 30 19:04:23.582142 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:23.582118 4862 scope.go:122] "RemoveContainer" containerID="8dcd1c44875aca8dcb2664248c47e79da149c8da011c0acd09e2121f3c95e10e" Oct 30 19:04:25.871082 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:25.871043 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:04:31.961752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:31.961721 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv"] Oct 30 19:04:31.962395 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:31.962379 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5f2a0513-e682-4a1e-aa20-c6f3e310ffab" containerName="revoke-keys" Oct 30 19:04:31.962455 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:31.962396 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2a0513-e682-4a1e-aa20-c6f3e310ffab" containerName="revoke-keys" Oct 30 19:04:31.962552 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:31.962543 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2a0513-e682-4a1e-aa20-c6f3e310ffab" containerName="revoke-keys" Oct 30 19:04:31.966117 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:31.966098 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:31.979694 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:31.979649 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:31.979980 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:31.979957 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:04:31.984548 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:31.984525 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv"] Oct 30 19:04:32.010487 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.010463 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003418fc-qvjbv\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.010643 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.010623 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003418fc-qvjbv\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.010759 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.010733 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbg8z\" (UniqueName: \"kubernetes.io/projected/3cceaf4e-429c-41a3-90f6-4f575a662229-kube-api-access-bbg8z\") pod \"maas-api-revoke-sub-003418fc-qvjbv\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.111817 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.111785 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003418fc-qvjbv\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.111992 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.111894 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003418fc-qvjbv\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.111992 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.111935 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbg8z\" (UniqueName: \"kubernetes.io/projected/3cceaf4e-429c-41a3-90f6-4f575a662229-kube-api-access-bbg8z\") pod \"maas-api-revoke-sub-003418fc-qvjbv\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.112661 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.112638 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003418fc-qvjbv\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.114514 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.114491 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003418fc-qvjbv\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.129514 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.129493 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbg8z\" (UniqueName: \"kubernetes.io/projected/3cceaf4e-429c-41a3-90f6-4f575a662229-kube-api-access-bbg8z\") pod \"maas-api-revoke-sub-003418fc-qvjbv\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.279673 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.279613 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:32.581125 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.581029 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv"] Oct 30 19:04:32.583969 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:04:32.583940 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cceaf4e_429c_41a3_90f6_4f575a662229.slice/crio-ff8e991f1efd1cb57dd3303c2842969fc28e1f5acecc2c6956d0c5f648a5c8ef WatchSource:0}: Error finding container ff8e991f1efd1cb57dd3303c2842969fc28e1f5acecc2c6956d0c5f648a5c8ef: Status 404 returned error can't find the container with id ff8e991f1efd1cb57dd3303c2842969fc28e1f5acecc2c6956d0c5f648a5c8ef Oct 30 19:04:32.653347 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:32.653317 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" event={"ID":"3cceaf4e-429c-41a3-90f6-4f575a662229","Type":"ContainerStarted","Data":"ff8e991f1efd1cb57dd3303c2842969fc28e1f5acecc2c6956d0c5f648a5c8ef"} Oct 30 19:04:33.662031 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:33.661946 4862 generic.go:356] "Generic (PLEG): container finished" podID="3cceaf4e-429c-41a3-90f6-4f575a662229" containerID="76add42882a48dec73facc3e7aff5bd347f7e0329e0a2a4bfa3149015f852782" exitCode=0 Oct 30 19:04:33.662440 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:33.662042 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" event={"ID":"3cceaf4e-429c-41a3-90f6-4f575a662229","Type":"ContainerDied","Data":"76add42882a48dec73facc3e7aff5bd347f7e0329e0a2a4bfa3149015f852782"} Oct 30 19:04:34.834538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:34.834499 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:34.950654 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:34.950575 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-xks-ca\") pod \"3cceaf4e-429c-41a3-90f6-4f575a662229\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " Oct 30 19:04:34.950654 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:34.950631 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cceaf4e-429c-41a3-90f6-4f575a662229-kube-api-access-bbg8z\" (UniqueName: \"kubernetes.io/projected/3cceaf4e-429c-41a3-90f6-4f575a662229-kube-api-access-bbg8z\") pod \"3cceaf4e-429c-41a3-90f6-4f575a662229\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " Oct 30 19:04:34.950892 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:34.950717 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-openshift-ca\") pod \"3cceaf4e-429c-41a3-90f6-4f575a662229\" (UID: \"3cceaf4e-429c-41a3-90f6-4f575a662229\") " Oct 30 19:04:34.951048 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:34.951025 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-openshift-ca" pod "3cceaf4e-429c-41a3-90f6-4f575a662229" (UID: "3cceaf4e-429c-41a3-90f6-4f575a662229"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:34.951404 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:34.951381 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:34.952991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:34.952969 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-xks-ca" pod "3cceaf4e-429c-41a3-90f6-4f575a662229" (UID: "3cceaf4e-429c-41a3-90f6-4f575a662229"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:34.953104 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:34.952991 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cceaf4e-429c-41a3-90f6-4f575a662229-kube-api-access-bbg8z" pod "3cceaf4e-429c-41a3-90f6-4f575a662229" (UID: "3cceaf4e-429c-41a3-90f6-4f575a662229"). InnerVolumeSpecName "kube-api-access-bbg8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:35.052223 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:35.052196 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cceaf4e-429c-41a3-90f6-4f575a662229-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:35.052223 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:35.052220 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbg8z\" (UniqueName: \"kubernetes.io/projected/3cceaf4e-429c-41a3-90f6-4f575a662229-kube-api-access-bbg8z\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:35.678646 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:35.678614 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" Oct 30 19:04:35.678646 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:35.678625 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv" event={"ID":"3cceaf4e-429c-41a3-90f6-4f575a662229","Type":"ContainerDied","Data":"ff8e991f1efd1cb57dd3303c2842969fc28e1f5acecc2c6956d0c5f648a5c8ef"} Oct 30 19:04:35.678876 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:35.678672 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8e991f1efd1cb57dd3303c2842969fc28e1f5acecc2c6956d0c5f648a5c8ef" Oct 30 19:04:36.443010 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.442975 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" podUID="16e9e7c8-5abc-448f-8dde-864c9238c181" containerName="main" containerID="cri-o://8007d855310410363d64389f8aecfa5b87604cea8e7e13ed2c8559a6b66d19ef" gracePeriod=45 Oct 30 19:04:36.492206 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.492169 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" podUID="16e9e7c8-5abc-448f-8dde-864c9238c181" containerName="main" probeResult="failure" output="Get \"https://10.133.0.238:8000/ready\": dial tcp 10.133.0.238:8000: connect: connection refused" Oct 30 19:04:36.578758 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.578724 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf"] Oct 30 19:04:36.579444 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.579422 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3cceaf4e-429c-41a3-90f6-4f575a662229" containerName="revoke-keys" Oct 30 19:04:36.579444 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.579440 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cceaf4e-429c-41a3-90f6-4f575a662229" containerName="revoke-keys" Oct 30 19:04:36.579643 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.579590 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cceaf4e-429c-41a3-90f6-4f575a662229" containerName="revoke-keys" Oct 30 19:04:36.583208 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.583184 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.602944 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.602916 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf"] Oct 30 19:04:36.608023 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.608000 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:36.608154 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.608111 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:04:36.673446 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.673416 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-112a46ec-vbkvf\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.673589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.673559 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8rk\" (UniqueName: \"kubernetes.io/projected/d8950842-58b0-4ac1-a6e4-780b719111fb-kube-api-access-wv8rk\") pod \"maas-api-revoke-sub-112a46ec-vbkvf\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.673718 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.673702 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-112a46ec-vbkvf\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.688120 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.688092 4862 generic.go:356] "Generic (PLEG): container finished" podID="16e9e7c8-5abc-448f-8dde-864c9238c181" containerID="8007d855310410363d64389f8aecfa5b87604cea8e7e13ed2c8559a6b66d19ef" exitCode=0 Oct 30 19:04:36.688277 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.688170 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" event={"ID":"16e9e7c8-5abc-448f-8dde-864c9238c181","Type":"ContainerDied","Data":"8007d855310410363d64389f8aecfa5b87604cea8e7e13ed2c8559a6b66d19ef"} Oct 30 19:04:36.765426 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.765400 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 19:04:36.775068 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.775040 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8rk\" (UniqueName: \"kubernetes.io/projected/d8950842-58b0-4ac1-a6e4-780b719111fb-kube-api-access-wv8rk\") pod \"maas-api-revoke-sub-112a46ec-vbkvf\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.775205 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.775155 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-112a46ec-vbkvf\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.775291 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.775232 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-112a46ec-vbkvf\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.775858 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.775834 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-112a46ec-vbkvf\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.777610 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.777585 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-112a46ec-vbkvf\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.792417 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.792393 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8rk\" (UniqueName: \"kubernetes.io/projected/d8950842-58b0-4ac1-a6e4-780b719111fb-kube-api-access-wv8rk\") pod \"maas-api-revoke-sub-112a46ec-vbkvf\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.875908 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.875879 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e9e7c8-5abc-448f-8dde-864c9238c181-kube-api-access-6gk99\" (UniqueName: \"kubernetes.io/projected/16e9e7c8-5abc-448f-8dde-864c9238c181-kube-api-access-6gk99\") pod \"16e9e7c8-5abc-448f-8dde-864c9238c181\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " Oct 30 19:04:36.876056 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.875955 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-home\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-home\") pod \"16e9e7c8-5abc-448f-8dde-864c9238c181\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " Oct 30 19:04:36.876056 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876024 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-model-cache\") pod \"16e9e7c8-5abc-448f-8dde-864c9238c181\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " Oct 30 19:04:36.876196 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876179 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-dshm\") pod \"16e9e7c8-5abc-448f-8dde-864c9238c181\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " Oct 30 19:04:36.876247 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876230 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-tmp-dir\") pod \"16e9e7c8-5abc-448f-8dde-864c9238c181\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " Oct 30 19:04:36.876321 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876277 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e9e7c8-5abc-448f-8dde-864c9238c181-tls-certs\" (UniqueName: \"kubernetes.io/secret/16e9e7c8-5abc-448f-8dde-864c9238c181-tls-certs\") pod \"16e9e7c8-5abc-448f-8dde-864c9238c181\" (UID: \"16e9e7c8-5abc-448f-8dde-864c9238c181\") " Oct 30 19:04:36.876321 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876299 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-home" pod "16e9e7c8-5abc-448f-8dde-864c9238c181" (UID: "16e9e7c8-5abc-448f-8dde-864c9238c181"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:36.876460 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876299 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-model-cache" pod "16e9e7c8-5abc-448f-8dde-864c9238c181" (UID: "16e9e7c8-5abc-448f-8dde-864c9238c181"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:36.876535 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876513 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-tmp-dir" pod "16e9e7c8-5abc-448f-8dde-864c9238c181" (UID: "16e9e7c8-5abc-448f-8dde-864c9238c181"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:36.876891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876873 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:36.876891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876892 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:36.877051 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.876901 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:36.878502 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.878481 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-dshm" pod "16e9e7c8-5abc-448f-8dde-864c9238c181" (UID: "16e9e7c8-5abc-448f-8dde-864c9238c181"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:36.878600 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.878530 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e9e7c8-5abc-448f-8dde-864c9238c181-tls-certs" pod "16e9e7c8-5abc-448f-8dde-864c9238c181" (UID: "16e9e7c8-5abc-448f-8dde-864c9238c181"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:36.878600 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.878559 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e9e7c8-5abc-448f-8dde-864c9238c181-kube-api-access-6gk99" pod "16e9e7c8-5abc-448f-8dde-864c9238c181" (UID: "16e9e7c8-5abc-448f-8dde-864c9238c181"). InnerVolumeSpecName "kube-api-access-6gk99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:36.901366 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.901343 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:36.978717 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.978689 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16e9e7c8-5abc-448f-8dde-864c9238c181-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:36.978717 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.978719 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e9e7c8-5abc-448f-8dde-864c9238c181-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:36.978875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:36.978730 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gk99\" (UniqueName: \"kubernetes.io/projected/16e9e7c8-5abc-448f-8dde-864c9238c181-kube-api-access-6gk99\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:37.204711 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:37.204687 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf"] Oct 30 19:04:37.207407 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:04:37.207381 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8950842_58b0_4ac1_a6e4_780b719111fb.slice/crio-918e151e19e228cea5bdbeb26a2e99bec18c961e94e7414e9206f4add3c072db WatchSource:0}: Error finding container 918e151e19e228cea5bdbeb26a2e99bec18c961e94e7414e9206f4add3c072db: Status 404 returned error can't find the container with id 918e151e19e228cea5bdbeb26a2e99bec18c961e94e7414e9206f4add3c072db Oct 30 19:04:37.699941 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:37.699864 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" event={"ID":"d8950842-58b0-4ac1-a6e4-780b719111fb","Type":"ContainerStarted","Data":"918e151e19e228cea5bdbeb26a2e99bec18c961e94e7414e9206f4add3c072db"} Oct 30 19:04:37.701475 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:37.701450 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" Oct 30 19:04:37.701608 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:37.701488 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj" event={"ID":"16e9e7c8-5abc-448f-8dde-864c9238c181","Type":"ContainerDied","Data":"ac655b97351669bf8fbbec6f7b561094f709818f56268bd2049fc8e369b76e8a"} Oct 30 19:04:37.701608 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:37.701543 4862 scope.go:122] "RemoveContainer" containerID="8007d855310410363d64389f8aecfa5b87604cea8e7e13ed2c8559a6b66d19ef" Oct 30 19:04:37.750527 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:37.750498 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj"] Oct 30 19:04:37.754325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:37.754305 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2820d9/test-model-2820d9-kserve-6944cb4bcc-728fj"] Oct 30 19:04:38.710138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:38.710103 4862 generic.go:356] "Generic (PLEG): container finished" podID="d8950842-58b0-4ac1-a6e4-780b719111fb" containerID="0e751a5d286604bcaa90565c8a48097ec8d676d61ccbe68e1c6ac7dec8bbf1d1" exitCode=0 Oct 30 19:04:38.710616 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:38.710196 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" event={"ID":"d8950842-58b0-4ac1-a6e4-780b719111fb","Type":"ContainerDied","Data":"0e751a5d286604bcaa90565c8a48097ec8d676d61ccbe68e1c6ac7dec8bbf1d1"} Oct 30 19:04:39.063525 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:39.063493 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e9e7c8-5abc-448f-8dde-864c9238c181" path="/var/lib/kubelet/pods/16e9e7c8-5abc-448f-8dde-864c9238c181/volumes" Oct 30 19:04:39.879090 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:39.879064 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:40.024712 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.024642 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8950842-58b0-4ac1-a6e4-780b719111fb-kube-api-access-wv8rk\" (UniqueName: \"kubernetes.io/projected/d8950842-58b0-4ac1-a6e4-780b719111fb-kube-api-access-wv8rk\") pod \"d8950842-58b0-4ac1-a6e4-780b719111fb\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " Oct 30 19:04:40.024868 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.024715 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-openshift-ca\") pod \"d8950842-58b0-4ac1-a6e4-780b719111fb\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " Oct 30 19:04:40.024868 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.024824 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-xks-ca\") pod \"d8950842-58b0-4ac1-a6e4-780b719111fb\" (UID: \"d8950842-58b0-4ac1-a6e4-780b719111fb\") " Oct 30 19:04:40.025065 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.025040 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-openshift-ca" pod "d8950842-58b0-4ac1-a6e4-780b719111fb" (UID: "d8950842-58b0-4ac1-a6e4-780b719111fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:40.025204 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.025192 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:40.026981 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.026952 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-xks-ca" pod "d8950842-58b0-4ac1-a6e4-780b719111fb" (UID: "d8950842-58b0-4ac1-a6e4-780b719111fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:40.027091 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.027011 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8950842-58b0-4ac1-a6e4-780b719111fb-kube-api-access-wv8rk" pod "d8950842-58b0-4ac1-a6e4-780b719111fb" (UID: "d8950842-58b0-4ac1-a6e4-780b719111fb"). InnerVolumeSpecName "kube-api-access-wv8rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:40.126013 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.125984 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8950842-58b0-4ac1-a6e4-780b719111fb-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:40.126013 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.126010 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv8rk\" (UniqueName: \"kubernetes.io/projected/d8950842-58b0-4ac1-a6e4-780b719111fb-kube-api-access-wv8rk\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:40.726499 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.726464 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" Oct 30 19:04:40.726712 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.726498 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf" event={"ID":"d8950842-58b0-4ac1-a6e4-780b719111fb","Type":"ContainerDied","Data":"918e151e19e228cea5bdbeb26a2e99bec18c961e94e7414e9206f4add3c072db"} Oct 30 19:04:40.726712 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:40.726550 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="918e151e19e228cea5bdbeb26a2e99bec18c961e94e7414e9206f4add3c072db" Oct 30 19:04:42.897457 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:42.897416 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk"] Oct 30 19:04:43.309469 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.309432 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v"] Oct 30 19:04:43.310141 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.310120 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d8950842-58b0-4ac1-a6e4-780b719111fb" containerName="revoke-keys" Oct 30 19:04:43.310141 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.310138 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8950842-58b0-4ac1-a6e4-780b719111fb" containerName="revoke-keys" Oct 30 19:04:43.310376 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.310150 4862 cpu_manager.go:414] "containerMap: removing container" podUID="16e9e7c8-5abc-448f-8dde-864c9238c181" containerName="main" Oct 30 19:04:43.310376 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.310155 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e9e7c8-5abc-448f-8dde-864c9238c181" containerName="main" Oct 30 19:04:43.310376 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.310322 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8950842-58b0-4ac1-a6e4-780b719111fb" containerName="revoke-keys" Oct 30 19:04:43.310376 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.310332 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e9e7c8-5abc-448f-8dde-864c9238c181" containerName="main" Oct 30 19:04:43.314422 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.314401 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.328301 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.328276 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:43.332280 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.332236 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v"] Oct 30 19:04:43.337675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.337653 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:04:43.380294 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.380231 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb5x4\" (UniqueName: \"kubernetes.io/projected/4fd89c88-5971-496f-991f-147e74ebca53-kube-api-access-lb5x4\") pod \"maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.380446 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.380429 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.380591 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.380576 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.482178 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.482148 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.482384 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.482289 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.482384 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.482333 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb5x4\" (UniqueName: \"kubernetes.io/projected/4fd89c88-5971-496f-991f-147e74ebca53-kube-api-access-lb5x4\") pod \"maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.482912 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.482891 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.485061 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.485042 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.499672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.499652 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb5x4\" (UniqueName: \"kubernetes.io/projected/4fd89c88-5971-496f-991f-147e74ebca53-kube-api-access-lb5x4\") pod \"maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.629139 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.629067 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:43.952560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:43.952532 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v"] Oct 30 19:04:43.954691 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:04:43.954662 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd89c88_5971_496f_991f_147e74ebca53.slice/crio-a49bcb16cc133fefad731c54e4896c02791ce5e17f0daf8a8441d24cc9e87fa1 WatchSource:0}: Error finding container a49bcb16cc133fefad731c54e4896c02791ce5e17f0daf8a8441d24cc9e87fa1: Status 404 returned error can't find the container with id a49bcb16cc133fefad731c54e4896c02791ce5e17f0daf8a8441d24cc9e87fa1 Oct 30 19:04:44.759275 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:44.759218 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" event={"ID":"4fd89c88-5971-496f-991f-147e74ebca53","Type":"ContainerStarted","Data":"f9a3b51dc019e4e899f3dd9eaaade35fc77822147a0783a0d6b0633e3addffb8"} Oct 30 19:04:44.759482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:44.759314 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" event={"ID":"4fd89c88-5971-496f-991f-147e74ebca53","Type":"ContainerStarted","Data":"a49bcb16cc133fefad731c54e4896c02791ce5e17f0daf8a8441d24cc9e87fa1"} Oct 30 19:04:44.796023 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:44.795976 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" podStartSLOduration=1.795962391 podStartE2EDuration="1.795962391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:44.792814679 +0000 UTC m=+3983.841859592" watchObservedRunningTime="2026-10-30 19:04:44.795962391 +0000 UTC m=+3983.845007304" Oct 30 19:04:45.769838 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:45.769804 4862 generic.go:356] "Generic (PLEG): container finished" podID="4fd89c88-5971-496f-991f-147e74ebca53" containerID="f9a3b51dc019e4e899f3dd9eaaade35fc77822147a0783a0d6b0633e3addffb8" exitCode=0 Oct 30 19:04:45.770220 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:45.769844 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" event={"ID":"4fd89c88-5971-496f-991f-147e74ebca53","Type":"ContainerDied","Data":"f9a3b51dc019e4e899f3dd9eaaade35fc77822147a0783a0d6b0633e3addffb8"} Oct 30 19:04:46.995709 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:46.995679 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:47.033672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.033606 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-openshift-ca\") pod \"4fd89c88-5971-496f-991f-147e74ebca53\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " Oct 30 19:04:47.033901 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.033860 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd89c88-5971-496f-991f-147e74ebca53-kube-api-access-lb5x4\" (UniqueName: \"kubernetes.io/projected/4fd89c88-5971-496f-991f-147e74ebca53-kube-api-access-lb5x4\") pod \"4fd89c88-5971-496f-991f-147e74ebca53\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " Oct 30 19:04:47.033901 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.033900 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-xks-ca\") pod \"4fd89c88-5971-496f-991f-147e74ebca53\" (UID: \"4fd89c88-5971-496f-991f-147e74ebca53\") " Oct 30 19:04:47.034163 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.033968 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-openshift-ca" pod "4fd89c88-5971-496f-991f-147e74ebca53" (UID: "4fd89c88-5971-496f-991f-147e74ebca53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:47.034618 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.034595 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:47.036610 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.036587 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd89c88-5971-496f-991f-147e74ebca53-kube-api-access-lb5x4" pod "4fd89c88-5971-496f-991f-147e74ebca53" (UID: "4fd89c88-5971-496f-991f-147e74ebca53"). InnerVolumeSpecName "kube-api-access-lb5x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:47.036748 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.036637 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-xks-ca" pod "4fd89c88-5971-496f-991f-147e74ebca53" (UID: "4fd89c88-5971-496f-991f-147e74ebca53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:47.136319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.136287 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb5x4\" (UniqueName: \"kubernetes.io/projected/4fd89c88-5971-496f-991f-147e74ebca53-kube-api-access-lb5x4\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:47.136561 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.136533 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd89c88-5971-496f-991f-147e74ebca53-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:47.792512 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.792480 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" Oct 30 19:04:47.792698 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.792505 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v" event={"ID":"4fd89c88-5971-496f-991f-147e74ebca53","Type":"ContainerDied","Data":"a49bcb16cc133fefad731c54e4896c02791ce5e17f0daf8a8441d24cc9e87fa1"} Oct 30 19:04:47.792698 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:47.792594 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a49bcb16cc133fefad731c54e4896c02791ce5e17f0daf8a8441d24cc9e87fa1" Oct 30 19:04:49.869957 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:04:49.869922 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:04:50.991358 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:50.991322 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b"] Oct 30 19:04:50.991989 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:50.991975 4862 cpu_manager.go:414] "containerMap: removing container" podUID="4fd89c88-5971-496f-991f-147e74ebca53" containerName="revoke-keys" Oct 30 19:04:50.992036 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:50.991990 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd89c88-5971-496f-991f-147e74ebca53" containerName="revoke-keys" Oct 30 19:04:50.992137 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:50.992128 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd89c88-5971-496f-991f-147e74ebca53" containerName="revoke-keys" Oct 30 19:04:50.995870 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:50.995852 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.013960 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.013936 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b"] Oct 30 19:04:51.020219 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.020194 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:51.020365 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.020244 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:04:51.139099 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.139061 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b60ae622-gsl5b\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.139304 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.139119 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b60ae622-gsl5b\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.139304 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.139210 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9kgm\" (UniqueName: \"kubernetes.io/projected/c2ce0962-1b98-4895-9fb2-c2377e33568d-kube-api-access-g9kgm\") pod \"maas-api-revoke-sub-b60ae622-gsl5b\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.241017 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.240992 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b60ae622-gsl5b\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.241173 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.241052 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9kgm\" (UniqueName: \"kubernetes.io/projected/c2ce0962-1b98-4895-9fb2-c2377e33568d-kube-api-access-g9kgm\") pod \"maas-api-revoke-sub-b60ae622-gsl5b\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.241503 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.241447 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b60ae622-gsl5b\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.241756 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.241736 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b60ae622-gsl5b\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.244012 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.243989 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b60ae622-gsl5b\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.258371 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.258340 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9kgm\" (UniqueName: \"kubernetes.io/projected/c2ce0962-1b98-4895-9fb2-c2377e33568d-kube-api-access-g9kgm\") pod \"maas-api-revoke-sub-b60ae622-gsl5b\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.309665 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.309643 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:51.613351 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.613322 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b"] Oct 30 19:04:51.613902 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:04:51.613852 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ce0962_1b98_4895_9fb2_c2377e33568d.slice/crio-8ddd629145e5facb6829438a346a0abcf1d226b1e4dde346966acd96076ae8a4 WatchSource:0}: Error finding container 8ddd629145e5facb6829438a346a0abcf1d226b1e4dde346966acd96076ae8a4: Status 404 returned error can't find the container with id 8ddd629145e5facb6829438a346a0abcf1d226b1e4dde346966acd96076ae8a4 Oct 30 19:04:51.822756 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:51.822726 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" event={"ID":"c2ce0962-1b98-4895-9fb2-c2377e33568d","Type":"ContainerStarted","Data":"8ddd629145e5facb6829438a346a0abcf1d226b1e4dde346966acd96076ae8a4"} Oct 30 19:04:52.831133 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:52.831099 4862 generic.go:356] "Generic (PLEG): container finished" podID="c2ce0962-1b98-4895-9fb2-c2377e33568d" containerID="57aca73380c7814b3901a3198af2ae38baffbe9519627ddaaa9c3674da28623d" exitCode=0 Oct 30 19:04:52.831536 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:52.831160 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" event={"ID":"c2ce0962-1b98-4895-9fb2-c2377e33568d","Type":"ContainerDied","Data":"57aca73380c7814b3901a3198af2ae38baffbe9519627ddaaa9c3674da28623d"} Oct 30 19:04:53.770860 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:53.770829 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p"] Oct 30 19:04:53.804205 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:53.804176 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq"] Oct 30 19:04:54.091834 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.091809 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:54.180134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.179909 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ce0962-1b98-4895-9fb2-c2377e33568d-kube-api-access-g9kgm\" (UniqueName: \"kubernetes.io/projected/c2ce0962-1b98-4895-9fb2-c2377e33568d-kube-api-access-g9kgm\") pod \"c2ce0962-1b98-4895-9fb2-c2377e33568d\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " Oct 30 19:04:54.180594 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.180567 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-xks-ca\") pod \"c2ce0962-1b98-4895-9fb2-c2377e33568d\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " Oct 30 19:04:54.180777 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.180634 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-openshift-ca\") pod \"c2ce0962-1b98-4895-9fb2-c2377e33568d\" (UID: \"c2ce0962-1b98-4895-9fb2-c2377e33568d\") " Oct 30 19:04:54.181881 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.181851 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-openshift-ca" pod "c2ce0962-1b98-4895-9fb2-c2377e33568d" (UID: "c2ce0962-1b98-4895-9fb2-c2377e33568d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:54.182775 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.182748 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ce0962-1b98-4895-9fb2-c2377e33568d-kube-api-access-g9kgm" pod "c2ce0962-1b98-4895-9fb2-c2377e33568d" (UID: "c2ce0962-1b98-4895-9fb2-c2377e33568d"). InnerVolumeSpecName "kube-api-access-g9kgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:54.183659 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.183635 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-xks-ca" pod "c2ce0962-1b98-4895-9fb2-c2377e33568d" (UID: "c2ce0962-1b98-4895-9fb2-c2377e33568d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:54.283177 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.283138 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9kgm\" (UniqueName: \"kubernetes.io/projected/c2ce0962-1b98-4895-9fb2-c2377e33568d-kube-api-access-g9kgm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:54.283177 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.283173 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:54.283466 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.283187 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ce0962-1b98-4895-9fb2-c2377e33568d-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:54.846478 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.846444 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" Oct 30 19:04:54.846651 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.846483 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b" event={"ID":"c2ce0962-1b98-4895-9fb2-c2377e33568d","Type":"ContainerDied","Data":"8ddd629145e5facb6829438a346a0abcf1d226b1e4dde346966acd96076ae8a4"} Oct 30 19:04:54.846651 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:54.846527 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ddd629145e5facb6829438a346a0abcf1d226b1e4dde346966acd96076ae8a4" Oct 30 19:04:55.834550 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.834512 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f"] Oct 30 19:04:55.835165 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.835151 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c2ce0962-1b98-4895-9fb2-c2377e33568d" containerName="revoke-keys" Oct 30 19:04:55.835211 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.835167 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ce0962-1b98-4895-9fb2-c2377e33568d" containerName="revoke-keys" Oct 30 19:04:55.835316 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.835304 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ce0962-1b98-4895-9fb2-c2377e33568d" containerName="revoke-keys" Oct 30 19:04:55.839050 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.839036 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:55.853815 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.853793 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:04:55.864513 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.864488 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:04:55.868770 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.868747 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f"] Oct 30 19:04:55.938216 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.938193 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzw5q\" (UniqueName: \"kubernetes.io/projected/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-kube-api-access-bzw5q\") pod \"maas-api-revoke-sub-e9190fff-fbd6f\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:55.938365 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.938253 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9190fff-fbd6f\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:55.938426 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:55.938402 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9190fff-fbd6f\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:56.039542 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.039517 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9190fff-fbd6f\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:56.039683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.039643 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzw5q\" (UniqueName: \"kubernetes.io/projected/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-kube-api-access-bzw5q\") pod \"maas-api-revoke-sub-e9190fff-fbd6f\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:56.039729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.039695 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9190fff-fbd6f\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:56.040355 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.040334 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9190fff-fbd6f\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:56.041794 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.041772 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9190fff-fbd6f\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:56.057472 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.057450 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzw5q\" (UniqueName: \"kubernetes.io/projected/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-kube-api-access-bzw5q\") pod \"maas-api-revoke-sub-e9190fff-fbd6f\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:56.153307 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.153240 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:56.462935 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.462905 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f"] Oct 30 19:04:56.464911 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:04:56.464878 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8590f4d0_5a3c_4058_b66c_2d1dcce6222b.slice/crio-2f3d56e2fb224e612ec1c17c656f84604c2caa7eb40de347367616e094874581 WatchSource:0}: Error finding container 2f3d56e2fb224e612ec1c17c656f84604c2caa7eb40de347367616e094874581: Status 404 returned error can't find the container with id 2f3d56e2fb224e612ec1c17c656f84604c2caa7eb40de347367616e094874581 Oct 30 19:04:56.863048 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.863014 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" event={"ID":"8590f4d0-5a3c-4058-b66c-2d1dcce6222b","Type":"ContainerStarted","Data":"7e379d82a602859e2776f7a3ab955017a03e8ccd7883b7d5e833527f960e120c"} Oct 30 19:04:56.863402 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.863077 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" event={"ID":"8590f4d0-5a3c-4058-b66c-2d1dcce6222b","Type":"ContainerStarted","Data":"2f3d56e2fb224e612ec1c17c656f84604c2caa7eb40de347367616e094874581"} Oct 30 19:04:56.899037 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:56.898996 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" podStartSLOduration=1.898983319 podStartE2EDuration="1.898983319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:56.897957681 +0000 UTC m=+3995.947002589" watchObservedRunningTime="2026-10-30 19:04:56.898983319 +0000 UTC m=+3995.948028234" Oct 30 19:04:57.871171 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:57.871135 4862 generic.go:356] "Generic (PLEG): container finished" podID="8590f4d0-5a3c-4058-b66c-2d1dcce6222b" containerID="7e379d82a602859e2776f7a3ab955017a03e8ccd7883b7d5e833527f960e120c" exitCode=0 Oct 30 19:04:57.871655 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:57.871217 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" event={"ID":"8590f4d0-5a3c-4058-b66c-2d1dcce6222b","Type":"ContainerDied","Data":"7e379d82a602859e2776f7a3ab955017a03e8ccd7883b7d5e833527f960e120c"} Oct 30 19:04:57.928321 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:57.928291 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" podUID="893e9668-5fdc-43b6-ab9d-b5e9d9f54410" containerName="main" containerID="cri-o://ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0" gracePeriod=45 Oct 30 19:04:58.217289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.217238 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 19:04:58.306801 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.306769 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-dshm\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-dshm\") pod \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " Oct 30 19:04:58.307088 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.307064 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tls-certs\" (UniqueName: \"kubernetes.io/secret/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tls-certs\") pod \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " Oct 30 19:04:58.307299 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.307239 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-kube-api-access-qfsj2\" (UniqueName: \"kubernetes.io/projected/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-kube-api-access-qfsj2\") pod \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " Oct 30 19:04:58.307793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.307421 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tmp-dir\") pod \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " Oct 30 19:04:58.307793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.307580 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-home\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-home\") pod \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " Oct 30 19:04:58.307793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.307628 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-model-cache\") pod \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\" (UID: \"893e9668-5fdc-43b6-ab9d-b5e9d9f54410\") " Oct 30 19:04:58.308105 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.307965 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tmp-dir" pod "893e9668-5fdc-43b6-ab9d-b5e9d9f54410" (UID: "893e9668-5fdc-43b6-ab9d-b5e9d9f54410"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:58.308324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.308248 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-home" pod "893e9668-5fdc-43b6-ab9d-b5e9d9f54410" (UID: "893e9668-5fdc-43b6-ab9d-b5e9d9f54410"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:58.309105 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.308626 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-model-cache" pod "893e9668-5fdc-43b6-ab9d-b5e9d9f54410" (UID: "893e9668-5fdc-43b6-ab9d-b5e9d9f54410"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:58.309105 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.309024 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:58.309105 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.309048 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:58.309105 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.309064 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:58.313225 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.313196 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-kube-api-access-qfsj2" pod "893e9668-5fdc-43b6-ab9d-b5e9d9f54410" (UID: "893e9668-5fdc-43b6-ab9d-b5e9d9f54410"). InnerVolumeSpecName "kube-api-access-qfsj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:58.321185 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.320830 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-dshm" pod "893e9668-5fdc-43b6-ab9d-b5e9d9f54410" (UID: "893e9668-5fdc-43b6-ab9d-b5e9d9f54410"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:58.328306 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.324471 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tls-certs" pod "893e9668-5fdc-43b6-ab9d-b5e9d9f54410" (UID: "893e9668-5fdc-43b6-ab9d-b5e9d9f54410"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:58.410683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.410650 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:58.410683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.410677 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:58.410683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.410689 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfsj2\" (UniqueName: \"kubernetes.io/projected/893e9668-5fdc-43b6-ab9d-b5e9d9f54410-kube-api-access-qfsj2\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:58.771735 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.771691 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" podUID="ab801c39-eeb3-40da-8353-2d7c9dab9181" containerName="payload-processing" containerID="cri-o://1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c" gracePeriod=25 Oct 30 19:04:58.804876 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.804843 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" podUID="d34f6da0-5ece-4140-8889-e6bea714f874" containerName="payload-pre-processing" containerID="cri-o://deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d" gracePeriod=25 Oct 30 19:04:58.882819 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.882779 4862 generic.go:356] "Generic (PLEG): container finished" podID="893e9668-5fdc-43b6-ab9d-b5e9d9f54410" containerID="ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0" exitCode=0 Oct 30 19:04:58.883248 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.883227 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" Oct 30 19:04:58.884630 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.884414 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" event={"ID":"893e9668-5fdc-43b6-ab9d-b5e9d9f54410","Type":"ContainerDied","Data":"ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0"} Oct 30 19:04:58.884630 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.884462 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk" event={"ID":"893e9668-5fdc-43b6-ab9d-b5e9d9f54410","Type":"ContainerDied","Data":"7ccdf12eadca837d38a513a2bfcfc1b16ed75799b52a5496945b89526798d2ef"} Oct 30 19:04:58.884630 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.884483 4862 scope.go:122] "RemoveContainer" containerID="ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0" Oct 30 19:04:58.899253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.899232 4862 scope.go:122] "RemoveContainer" containerID="ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0" Oct 30 19:04:58.899622 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:04:58.899593 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0\": container with ID starting with ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0 not found: ID does not exist" containerID="ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0" Oct 30 19:04:58.899738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.899636 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0"} err="failed to get container status \"ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0\": rpc error: code = NotFound desc = could not find container \"ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0\": container with ID starting with ce98b934eed81c53f125a5ca2e4a879468281131145f64d011514247c28494e0 not found: ID does not exist" Oct 30 19:04:58.937405 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.937352 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk"] Oct 30 19:04:58.947851 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:58.947769 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9f6b7c/test-model-9f6b7c-kserve-6d6d64489c-kgqlk"] Oct 30 19:04:59.139290 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.127736 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893e9668-5fdc-43b6-ab9d-b5e9d9f54410" path="/var/lib/kubelet/pods/893e9668-5fdc-43b6-ab9d-b5e9d9f54410/volumes" Oct 30 19:04:59.295227 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.295197 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 19:04:59.375198 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.375160 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab801c39-eeb3-40da-8353-2d7c9dab9181-kube-api-access-9qxwl\" (UniqueName: \"kubernetes.io/projected/ab801c39-eeb3-40da-8353-2d7c9dab9181-kube-api-access-9qxwl\") pod \"ab801c39-eeb3-40da-8353-2d7c9dab9181\" (UID: \"ab801c39-eeb3-40da-8353-2d7c9dab9181\") " Oct 30 19:04:59.375492 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.375362 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab801c39-eeb3-40da-8353-2d7c9dab9181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab801c39-eeb3-40da-8353-2d7c9dab9181-plugins-config-volume\") pod \"ab801c39-eeb3-40da-8353-2d7c9dab9181\" (UID: \"ab801c39-eeb3-40da-8353-2d7c9dab9181\") " Oct 30 19:04:59.375876 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.375837 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab801c39-eeb3-40da-8353-2d7c9dab9181-plugins-config-volume" pod "ab801c39-eeb3-40da-8353-2d7c9dab9181" (UID: "ab801c39-eeb3-40da-8353-2d7c9dab9181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:59.376155 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.376131 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab801c39-eeb3-40da-8353-2d7c9dab9181-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:59.379682 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.379650 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab801c39-eeb3-40da-8353-2d7c9dab9181-kube-api-access-9qxwl" pod "ab801c39-eeb3-40da-8353-2d7c9dab9181" (UID: "ab801c39-eeb3-40da-8353-2d7c9dab9181"). InnerVolumeSpecName "kube-api-access-9qxwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:59.419512 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.419482 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:59.476673 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.476645 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-kube-api-access-bzw5q\" (UniqueName: \"kubernetes.io/projected/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-kube-api-access-bzw5q\") pod \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " Oct 30 19:04:59.476838 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.476765 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-openshift-ca\") pod \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " Oct 30 19:04:59.476838 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.476812 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-xks-ca\") pod \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\" (UID: \"8590f4d0-5a3c-4058-b66c-2d1dcce6222b\") " Oct 30 19:04:59.477155 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.477126 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-openshift-ca" pod "8590f4d0-5a3c-4058-b66c-2d1dcce6222b" (UID: "8590f4d0-5a3c-4058-b66c-2d1dcce6222b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:04:59.477943 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.477915 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qxwl\" (UniqueName: \"kubernetes.io/projected/ab801c39-eeb3-40da-8353-2d7c9dab9181-kube-api-access-9qxwl\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:59.478080 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.477947 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:59.479228 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.479199 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-kube-api-access-bzw5q" pod "8590f4d0-5a3c-4058-b66c-2d1dcce6222b" (UID: "8590f4d0-5a3c-4058-b66c-2d1dcce6222b"). InnerVolumeSpecName "kube-api-access-bzw5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:59.479398 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.479341 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-xks-ca" pod "8590f4d0-5a3c-4058-b66c-2d1dcce6222b" (UID: "8590f4d0-5a3c-4058-b66c-2d1dcce6222b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:59.579397 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.579364 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzw5q\" (UniqueName: \"kubernetes.io/projected/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-kube-api-access-bzw5q\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:59.579397 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.579399 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8590f4d0-5a3c-4058-b66c-2d1dcce6222b-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:04:59.893314 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.893221 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" Oct 30 19:04:59.893314 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.893275 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f" event={"ID":"8590f4d0-5a3c-4058-b66c-2d1dcce6222b","Type":"ContainerDied","Data":"2f3d56e2fb224e612ec1c17c656f84604c2caa7eb40de347367616e094874581"} Oct 30 19:04:59.893844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.893323 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f3d56e2fb224e612ec1c17c656f84604c2caa7eb40de347367616e094874581" Oct 30 19:04:59.894784 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.894754 4862 generic.go:356] "Generic (PLEG): container finished" podID="ab801c39-eeb3-40da-8353-2d7c9dab9181" containerID="1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c" exitCode=0 Oct 30 19:04:59.894924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.894810 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" Oct 30 19:04:59.894924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.894841 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" event={"ID":"ab801c39-eeb3-40da-8353-2d7c9dab9181","Type":"ContainerDied","Data":"1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c"} Oct 30 19:04:59.894924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.894887 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p" event={"ID":"ab801c39-eeb3-40da-8353-2d7c9dab9181","Type":"ContainerDied","Data":"a5b0620d8bf8f816ddd2ef9d7e991c2a429987eafb58410bc1faf440734e9dfd"} Oct 30 19:04:59.894924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.894910 4862 scope.go:122] "RemoveContainer" containerID="1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c" Oct 30 19:04:59.919652 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.918486 4862 scope.go:122] "RemoveContainer" containerID="1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c" Oct 30 19:04:59.919652 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:04:59.919554 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c\": container with ID starting with 1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c not found: ID does not exist" containerID="1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c" Oct 30 19:04:59.919652 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.919587 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c"} err="failed to get container status \"1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c\": rpc error: code = NotFound desc = could not find container \"1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c\": container with ID starting with 1e4acb3fb27fb90711d1155f8806f7b0ade423bcd18b39e1289a62a213fc6c8c not found: ID does not exist" Oct 30 19:04:59.957858 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.957830 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p"] Oct 30 19:04:59.964662 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:04:59.964637 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2820d9-787bf5fb78-bh47p"] Oct 30 19:05:01.064003 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:01.063967 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab801c39-eeb3-40da-8353-2d7c9dab9181" path="/var/lib/kubelet/pods/ab801c39-eeb3-40da-8353-2d7c9dab9181/volumes" Oct 30 19:05:02.460130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:02.460099 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" podUID="d34f6da0-5ece-4140-8889-e6bea714f874" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:05:11.650212 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:11.650178 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:05:12.497689 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:12.497449 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" podUID="d34f6da0-5ece-4140-8889-e6bea714f874" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:05:14.724676 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.724645 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6"] Oct 30 19:05:14.725317 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.725299 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ab801c39-eeb3-40da-8353-2d7c9dab9181" containerName="payload-processing" Oct 30 19:05:14.725317 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.725318 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab801c39-eeb3-40da-8353-2d7c9dab9181" containerName="payload-processing" Oct 30 19:05:14.725410 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.725349 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8590f4d0-5a3c-4058-b66c-2d1dcce6222b" containerName="revoke-keys" Oct 30 19:05:14.725410 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.725355 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8590f4d0-5a3c-4058-b66c-2d1dcce6222b" containerName="revoke-keys" Oct 30 19:05:14.725410 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.725370 4862 cpu_manager.go:414] "containerMap: removing container" podUID="893e9668-5fdc-43b6-ab9d-b5e9d9f54410" containerName="main" Oct 30 19:05:14.725410 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.725374 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893e9668-5fdc-43b6-ab9d-b5e9d9f54410" containerName="main" Oct 30 19:05:14.725538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.725482 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab801c39-eeb3-40da-8353-2d7c9dab9181" containerName="payload-processing" Oct 30 19:05:14.725538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.725495 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8590f4d0-5a3c-4058-b66c-2d1dcce6222b" containerName="revoke-keys" Oct 30 19:05:14.725538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.725502 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="893e9668-5fdc-43b6-ab9d-b5e9d9f54410" containerName="main" Oct 30 19:05:14.737285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.734805 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:14.802283 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.796599 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-15e8827b-dockercfg-pg257\"" Oct 30 19:05:14.802283 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.796661 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-15e8827b\"" Oct 30 19:05:14.896231 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.895838 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b124a84-203f-4628-96bd-4691396ffafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6\" (UID: \"0b124a84-203f-4628-96bd-4691396ffafc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:14.896231 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.896060 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztj74\" (UniqueName: \"kubernetes.io/projected/0b124a84-203f-4628-96bd-4691396ffafc-kube-api-access-ztj74\") pod \"payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6\" (UID: \"0b124a84-203f-4628-96bd-4691396ffafc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:14.930285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:14.929634 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6"] Oct 30 19:05:15.027343 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.027007 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b124a84-203f-4628-96bd-4691396ffafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6\" (UID: \"0b124a84-203f-4628-96bd-4691396ffafc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:15.027343 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.027246 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztj74\" (UniqueName: \"kubernetes.io/projected/0b124a84-203f-4628-96bd-4691396ffafc-kube-api-access-ztj74\") pod \"payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6\" (UID: \"0b124a84-203f-4628-96bd-4691396ffafc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:15.122953 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.122846 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b124a84-203f-4628-96bd-4691396ffafc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6\" (UID: \"0b124a84-203f-4628-96bd-4691396ffafc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:15.188286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.178969 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztj74\" (UniqueName: \"kubernetes.io/projected/0b124a84-203f-4628-96bd-4691396ffafc-kube-api-access-ztj74\") pod \"payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6\" (UID: \"0b124a84-203f-4628-96bd-4691396ffafc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:15.293979 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.293892 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb"] Oct 30 19:05:15.300066 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.300042 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb"] Oct 30 19:05:15.300368 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.300350 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:15.316896 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.316869 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:15.327648 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.327620 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa302575-37dc-4227-a316-56258980eb26-plugins-config-volume\") pod \"payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb\" (UID: \"fa302575-37dc-4227-a316-56258980eb26\") " pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:15.327827 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.327809 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54bvv\" (UniqueName: \"kubernetes.io/projected/fa302575-37dc-4227-a316-56258980eb26-kube-api-access-54bvv\") pod \"payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb\" (UID: \"fa302575-37dc-4227-a316-56258980eb26\") " pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:15.429407 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.429381 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa302575-37dc-4227-a316-56258980eb26-plugins-config-volume\") pod \"payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb\" (UID: \"fa302575-37dc-4227-a316-56258980eb26\") " pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:15.429644 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.429622 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54bvv\" (UniqueName: \"kubernetes.io/projected/fa302575-37dc-4227-a316-56258980eb26-kube-api-access-54bvv\") pod \"payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb\" (UID: \"fa302575-37dc-4227-a316-56258980eb26\") " pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:15.430384 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.430349 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa302575-37dc-4227-a316-56258980eb26-plugins-config-volume\") pod \"payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb\" (UID: \"fa302575-37dc-4227-a316-56258980eb26\") " pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:15.473282 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.456044 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54bvv\" (UniqueName: \"kubernetes.io/projected/fa302575-37dc-4227-a316-56258980eb26-kube-api-access-54bvv\") pod \"payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb\" (UID: \"fa302575-37dc-4227-a316-56258980eb26\") " pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:15.657794 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.657748 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:15.763082 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.762910 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6"] Oct 30 19:05:15.766129 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:05:15.766095 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b124a84_203f_4628_96bd_4691396ffafc.slice/crio-d92405e7ad1759f89625c3961eebb49d103a46935abc2cb364e0638dba3606b5 WatchSource:0}: Error finding container d92405e7ad1759f89625c3961eebb49d103a46935abc2cb364e0638dba3606b5: Status 404 returned error can't find the container with id d92405e7ad1759f89625c3961eebb49d103a46935abc2cb364e0638dba3606b5 Oct 30 19:05:15.959546 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:15.959518 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb"] Oct 30 19:05:15.962017 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:05:15.961990 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa302575_37dc_4227_a316_56258980eb26.slice/crio-5cdf93e20bda6b212c031aea7d20ba3326de5e2c45aabbbab6c64ca356c9a7f5 WatchSource:0}: Error finding container 5cdf93e20bda6b212c031aea7d20ba3326de5e2c45aabbbab6c64ca356c9a7f5: Status 404 returned error can't find the container with id 5cdf93e20bda6b212c031aea7d20ba3326de5e2c45aabbbab6c64ca356c9a7f5 Oct 30 19:05:16.102678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:16.102646 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" event={"ID":"fa302575-37dc-4227-a316-56258980eb26","Type":"ContainerStarted","Data":"71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a"} Oct 30 19:05:16.102818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:16.102705 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" event={"ID":"fa302575-37dc-4227-a316-56258980eb26","Type":"ContainerStarted","Data":"5cdf93e20bda6b212c031aea7d20ba3326de5e2c45aabbbab6c64ca356c9a7f5"} Oct 30 19:05:16.102818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:16.102751 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:16.104470 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:16.104445 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" event={"ID":"0b124a84-203f-4628-96bd-4691396ffafc","Type":"ContainerStarted","Data":"5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd"} Oct 30 19:05:16.104559 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:16.104487 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" event={"ID":"0b124a84-203f-4628-96bd-4691396ffafc","Type":"ContainerStarted","Data":"d92405e7ad1759f89625c3961eebb49d103a46935abc2cb364e0638dba3606b5"} Oct 30 19:05:16.104559 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:16.104538 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:16.138810 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:16.138763 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" podStartSLOduration=2.138745679 podStartE2EDuration="2.138745679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:16.13724442 +0000 UTC m=+4015.186289334" watchObservedRunningTime="2026-10-30 19:05:16.138745679 +0000 UTC m=+4015.187790597" Oct 30 19:05:16.174840 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:16.174797 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" podStartSLOduration=2.174784634 podStartE2EDuration="2.174784634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:16.171464424 +0000 UTC m=+4015.220509338" watchObservedRunningTime="2026-10-30 19:05:16.174784634 +0000 UTC m=+4015.223829546" Oct 30 19:05:17.184429 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.184396 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd"] Oct 30 19:05:17.188806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.188789 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.202928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.202905 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:05:17.213611 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.213586 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:05:17.217574 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.217554 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd"] Oct 30 19:05:17.261079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.261057 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.261232 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.261213 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74hzf\" (UniqueName: \"kubernetes.io/projected/5e988047-9f02-4e29-a70d-25cfe763afc5-kube-api-access-74hzf\") pod \"maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.261344 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.261329 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.362616 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.362590 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74hzf\" (UniqueName: \"kubernetes.io/projected/5e988047-9f02-4e29-a70d-25cfe763afc5-kube-api-access-74hzf\") pod \"maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.362768 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.362696 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.362857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.362841 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.363332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.363315 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.365335 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.365316 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.381403 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.381382 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74hzf\" (UniqueName: \"kubernetes.io/projected/5e988047-9f02-4e29-a70d-25cfe763afc5-kube-api-access-74hzf\") pod \"maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.504996 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.504943 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:17.821381 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:17.821354 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd"] Oct 30 19:05:17.821908 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:05:17.821876 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e988047_9f02_4e29_a70d_25cfe763afc5.slice/crio-a6b8780e0142f4155f93b74b4431ec488e60ca2f629d7bdb0b022b2abd9ebd5c WatchSource:0}: Error finding container a6b8780e0142f4155f93b74b4431ec488e60ca2f629d7bdb0b022b2abd9ebd5c: Status 404 returned error can't find the container with id a6b8780e0142f4155f93b74b4431ec488e60ca2f629d7bdb0b022b2abd9ebd5c Oct 30 19:05:18.120762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:18.120698 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" event={"ID":"5e988047-9f02-4e29-a70d-25cfe763afc5","Type":"ContainerStarted","Data":"a6b8780e0142f4155f93b74b4431ec488e60ca2f629d7bdb0b022b2abd9ebd5c"} Oct 30 19:05:19.094236 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.094205 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 19:05:19.129405 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.129375 4862 generic.go:356] "Generic (PLEG): container finished" podID="5e988047-9f02-4e29-a70d-25cfe763afc5" containerID="73146b833099fc6054e74b19293aa2a0cb5bbaf15716cbee5ca9268a59191f30" exitCode=0 Oct 30 19:05:19.129535 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.129457 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" event={"ID":"5e988047-9f02-4e29-a70d-25cfe763afc5","Type":"ContainerDied","Data":"73146b833099fc6054e74b19293aa2a0cb5bbaf15716cbee5ca9268a59191f30"} Oct 30 19:05:19.130731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.130707 4862 generic.go:356] "Generic (PLEG): container finished" podID="d34f6da0-5ece-4140-8889-e6bea714f874" containerID="deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d" exitCode=0 Oct 30 19:05:19.130828 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.130741 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" Oct 30 19:05:19.130828 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.130754 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" event={"ID":"d34f6da0-5ece-4140-8889-e6bea714f874","Type":"ContainerDied","Data":"deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d"} Oct 30 19:05:19.130828 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.130788 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq" event={"ID":"d34f6da0-5ece-4140-8889-e6bea714f874","Type":"ContainerDied","Data":"db3f9c0f33253a29000130eb01ff839c6493587d0be2d14d25ba5bb670e4d4da"} Oct 30 19:05:19.130828 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.130808 4862 scope.go:122] "RemoveContainer" containerID="deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d" Oct 30 19:05:19.148887 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.147239 4862 scope.go:122] "RemoveContainer" containerID="deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d" Oct 30 19:05:19.148887 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:05:19.148167 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d\": container with ID starting with deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d not found: ID does not exist" containerID="deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d" Oct 30 19:05:19.148887 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.148199 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d"} err="failed to get container status \"deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d\": rpc error: code = NotFound desc = could not find container \"deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d\": container with ID starting with deb3f274177a083d6f737fe1e731195b07188bfc11cc3af33c7707b688ac306d not found: ID does not exist" Oct 30 19:05:19.188254 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.188183 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34f6da0-5ece-4140-8889-e6bea714f874-kube-api-access-f6b4z\" (UniqueName: \"kubernetes.io/projected/d34f6da0-5ece-4140-8889-e6bea714f874-kube-api-access-f6b4z\") pod \"d34f6da0-5ece-4140-8889-e6bea714f874\" (UID: \"d34f6da0-5ece-4140-8889-e6bea714f874\") " Oct 30 19:05:19.188459 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.188296 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34f6da0-5ece-4140-8889-e6bea714f874-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34f6da0-5ece-4140-8889-e6bea714f874-plugins-config-volume\") pod \"d34f6da0-5ece-4140-8889-e6bea714f874\" (UID: \"d34f6da0-5ece-4140-8889-e6bea714f874\") " Oct 30 19:05:19.188649 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.188626 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34f6da0-5ece-4140-8889-e6bea714f874-plugins-config-volume" pod "d34f6da0-5ece-4140-8889-e6bea714f874" (UID: "d34f6da0-5ece-4140-8889-e6bea714f874"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:19.188825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.188810 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d34f6da0-5ece-4140-8889-e6bea714f874-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:19.190756 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.190727 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34f6da0-5ece-4140-8889-e6bea714f874-kube-api-access-f6b4z" pod "d34f6da0-5ece-4140-8889-e6bea714f874" (UID: "d34f6da0-5ece-4140-8889-e6bea714f874"). InnerVolumeSpecName "kube-api-access-f6b4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:19.289933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.289904 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6b4z\" (UniqueName: \"kubernetes.io/projected/d34f6da0-5ece-4140-8889-e6bea714f874-kube-api-access-f6b4z\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:19.477947 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.477916 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq"] Oct 30 19:05:19.488098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:19.488071 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2820d9-585c885cb7-t98wq"] Oct 30 19:05:19.871629 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:05:19.871590 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:05:20.334560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.334535 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:20.404862 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.404786 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e988047-9f02-4e29-a70d-25cfe763afc5-kube-api-access-74hzf\" (UniqueName: \"kubernetes.io/projected/5e988047-9f02-4e29-a70d-25cfe763afc5-kube-api-access-74hzf\") pod \"5e988047-9f02-4e29-a70d-25cfe763afc5\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " Oct 30 19:05:20.405035 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.404995 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-openshift-ca\") pod \"5e988047-9f02-4e29-a70d-25cfe763afc5\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " Oct 30 19:05:20.405129 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.405074 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-xks-ca\") pod \"5e988047-9f02-4e29-a70d-25cfe763afc5\" (UID: \"5e988047-9f02-4e29-a70d-25cfe763afc5\") " Oct 30 19:05:20.405453 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.405424 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-openshift-ca" pod "5e988047-9f02-4e29-a70d-25cfe763afc5" (UID: "5e988047-9f02-4e29-a70d-25cfe763afc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:20.405725 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.405705 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:20.409328 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.409300 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e988047-9f02-4e29-a70d-25cfe763afc5-kube-api-access-74hzf" pod "5e988047-9f02-4e29-a70d-25cfe763afc5" (UID: "5e988047-9f02-4e29-a70d-25cfe763afc5"). InnerVolumeSpecName "kube-api-access-74hzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:20.413620 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.413594 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-xks-ca" pod "5e988047-9f02-4e29-a70d-25cfe763afc5" (UID: "5e988047-9f02-4e29-a70d-25cfe763afc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:20.507116 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.507095 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74hzf\" (UniqueName: \"kubernetes.io/projected/5e988047-9f02-4e29-a70d-25cfe763afc5-kube-api-access-74hzf\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:20.507209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:20.507119 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e988047-9f02-4e29-a70d-25cfe763afc5-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:21.063182 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:21.063153 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34f6da0-5ece-4140-8889-e6bea714f874" path="/var/lib/kubelet/pods/d34f6da0-5ece-4140-8889-e6bea714f874/volumes" Oct 30 19:05:21.149746 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:21.149717 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" Oct 30 19:05:21.149889 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:21.149740 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd" event={"ID":"5e988047-9f02-4e29-a70d-25cfe763afc5","Type":"ContainerDied","Data":"a6b8780e0142f4155f93b74b4431ec488e60ca2f629d7bdb0b022b2abd9ebd5c"} Oct 30 19:05:21.149889 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:21.149793 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6b8780e0142f4155f93b74b4431ec488e60ca2f629d7bdb0b022b2abd9ebd5c" Oct 30 19:05:27.115583 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:27.115554 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:05:27.116104 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:27.115814 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:05:27.690604 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:27.690570 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v"] Oct 30 19:05:32.691154 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:32.691096 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" podUID="2b0ab5dd-725d-4c8c-9530-311383a99f96" containerName="payload-processing" containerID="cri-o://ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad" gracePeriod=25 Oct 30 19:05:37.271347 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:37.271312 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd"] Oct 30 19:05:37.277049 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:37.277016 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-869b55-nvmnd"] Oct 30 19:05:38.096757 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:38.096720 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" podUID="2b0ab5dd-725d-4c8c-9530-311383a99f96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:05:39.065135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:39.065091 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e988047-9f02-4e29-a70d-25cfe763afc5" path="/var/lib/kubelet/pods/5e988047-9f02-4e29-a70d-25cfe763afc5/volumes" Oct 30 19:05:45.114338 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.114291 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467"] Oct 30 19:05:45.115153 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.115130 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d34f6da0-5ece-4140-8889-e6bea714f874" containerName="payload-pre-processing" Oct 30 19:05:45.115304 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.115154 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34f6da0-5ece-4140-8889-e6bea714f874" containerName="payload-pre-processing" Oct 30 19:05:45.115304 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.115190 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5e988047-9f02-4e29-a70d-25cfe763afc5" containerName="revoke-keys" Oct 30 19:05:45.115304 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.115199 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e988047-9f02-4e29-a70d-25cfe763afc5" containerName="revoke-keys" Oct 30 19:05:45.115488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.115396 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34f6da0-5ece-4140-8889-e6bea714f874" containerName="payload-pre-processing" Oct 30 19:05:45.115488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.115417 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e988047-9f02-4e29-a70d-25cfe763afc5" containerName="revoke-keys" Oct 30 19:05:45.120521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.120500 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.138957 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.138926 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467"] Oct 30 19:05:45.180826 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.180796 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:05:45.180975 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.180892 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:05:45.204883 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.204859 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.205009 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.204909 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkmf\" (UniqueName: \"kubernetes.io/projected/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-kube-api-access-9hkmf\") pod \"maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.205107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.205062 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.306107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.306082 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.306245 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.306134 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkmf\" (UniqueName: \"kubernetes.io/projected/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-kube-api-access-9hkmf\") pod \"maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.306245 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.306216 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.306878 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.306860 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.308594 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.308572 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.324041 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.324009 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkmf\" (UniqueName: \"kubernetes.io/projected/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-kube-api-access-9hkmf\") pod \"maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.434920 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.434858 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:45.736364 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:45.736217 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467"] Oct 30 19:05:45.739707 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:05:45.739681 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8503581d_12f9_4af9_a9eb_5bc89d5bde5a.slice/crio-34e9dac7f4d7c3ed1dbc80a4234a02cb0eff18e611bf9f76859372ee668bb7da WatchSource:0}: Error finding container 34e9dac7f4d7c3ed1dbc80a4234a02cb0eff18e611bf9f76859372ee668bb7da: Status 404 returned error can't find the container with id 34e9dac7f4d7c3ed1dbc80a4234a02cb0eff18e611bf9f76859372ee668bb7da Oct 30 19:05:46.353063 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:46.353024 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" event={"ID":"8503581d-12f9-4af9-a9eb-5bc89d5bde5a","Type":"ContainerStarted","Data":"d37781471f11269e1bbca8e34694c0d43e9da3df4ef741054fef315a75cd9914"} Oct 30 19:05:46.353591 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:46.353095 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" event={"ID":"8503581d-12f9-4af9-a9eb-5bc89d5bde5a","Type":"ContainerStarted","Data":"34e9dac7f4d7c3ed1dbc80a4234a02cb0eff18e611bf9f76859372ee668bb7da"} Oct 30 19:05:46.397985 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:46.397921 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" podStartSLOduration=1.397899429 podStartE2EDuration="1.397899429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:46.395685395 +0000 UTC m=+4045.444730308" watchObservedRunningTime="2026-10-30 19:05:46.397899429 +0000 UTC m=+4045.446944343" Oct 30 19:05:47.337993 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:47.337962 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v"] Oct 30 19:05:47.347806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:47.347780 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2820d9-vc75v"] Oct 30 19:05:47.420817 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:47.420776 4862 generic.go:356] "Generic (PLEG): container finished" podID="8503581d-12f9-4af9-a9eb-5bc89d5bde5a" containerID="d37781471f11269e1bbca8e34694c0d43e9da3df4ef741054fef315a75cd9914" exitCode=0 Oct 30 19:05:47.421329 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:47.420826 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" event={"ID":"8503581d-12f9-4af9-a9eb-5bc89d5bde5a","Type":"ContainerDied","Data":"d37781471f11269e1bbca8e34694c0d43e9da3df4ef741054fef315a75cd9914"} Oct 30 19:05:48.096036 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.095995 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" podUID="2b0ab5dd-725d-4c8c-9530-311383a99f96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:05:48.589188 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.589163 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:48.659596 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.659560 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-xks-ca\") pod \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " Oct 30 19:05:48.659854 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.659831 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-openshift-ca\") pod \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " Oct 30 19:05:48.659942 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.659893 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-kube-api-access-9hkmf\" (UniqueName: \"kubernetes.io/projected/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-kube-api-access-9hkmf\") pod \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\" (UID: \"8503581d-12f9-4af9-a9eb-5bc89d5bde5a\") " Oct 30 19:05:48.660140 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.660112 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-openshift-ca" pod "8503581d-12f9-4af9-a9eb-5bc89d5bde5a" (UID: "8503581d-12f9-4af9-a9eb-5bc89d5bde5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:48.660672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.660609 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:48.662159 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.662132 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-xks-ca" pod "8503581d-12f9-4af9-a9eb-5bc89d5bde5a" (UID: "8503581d-12f9-4af9-a9eb-5bc89d5bde5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:48.662342 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.662199 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-kube-api-access-9hkmf" pod "8503581d-12f9-4af9-a9eb-5bc89d5bde5a" (UID: "8503581d-12f9-4af9-a9eb-5bc89d5bde5a"). InnerVolumeSpecName "kube-api-access-9hkmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:48.761780 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.761708 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:48.761780 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:48.761736 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hkmf\" (UniqueName: \"kubernetes.io/projected/8503581d-12f9-4af9-a9eb-5bc89d5bde5a-kube-api-access-9hkmf\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:49.063564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:49.063533 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd89c88-5971-496f-991f-147e74ebca53" path="/var/lib/kubelet/pods/4fd89c88-5971-496f-991f-147e74ebca53/volumes" Oct 30 19:05:49.436702 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:49.436624 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" Oct 30 19:05:49.436702 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:49.436659 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467" event={"ID":"8503581d-12f9-4af9-a9eb-5bc89d5bde5a","Type":"ContainerDied","Data":"34e9dac7f4d7c3ed1dbc80a4234a02cb0eff18e611bf9f76859372ee668bb7da"} Oct 30 19:05:49.436910 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:49.436718 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e9dac7f4d7c3ed1dbc80a4234a02cb0eff18e611bf9f76859372ee668bb7da" Oct 30 19:05:49.872944 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:05:49.872897 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:05:53.011846 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.011819 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 19:05:53.112322 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.109061 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0ab5dd-725d-4c8c-9530-311383a99f96-kube-api-access-7f5tj\" (UniqueName: \"kubernetes.io/projected/2b0ab5dd-725d-4c8c-9530-311383a99f96-kube-api-access-7f5tj\") pod \"2b0ab5dd-725d-4c8c-9530-311383a99f96\" (UID: \"2b0ab5dd-725d-4c8c-9530-311383a99f96\") " Oct 30 19:05:53.112322 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.109388 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0ab5dd-725d-4c8c-9530-311383a99f96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0ab5dd-725d-4c8c-9530-311383a99f96-plugins-config-volume\") pod \"2b0ab5dd-725d-4c8c-9530-311383a99f96\" (UID: \"2b0ab5dd-725d-4c8c-9530-311383a99f96\") " Oct 30 19:05:53.119223 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.116670 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0ab5dd-725d-4c8c-9530-311383a99f96-plugins-config-volume" pod "2b0ab5dd-725d-4c8c-9530-311383a99f96" (UID: "2b0ab5dd-725d-4c8c-9530-311383a99f96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:53.123335 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.123243 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0ab5dd-725d-4c8c-9530-311383a99f96-kube-api-access-7f5tj" pod "2b0ab5dd-725d-4c8c-9530-311383a99f96" (UID: "2b0ab5dd-725d-4c8c-9530-311383a99f96"). InnerVolumeSpecName "kube-api-access-7f5tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:53.211578 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.211543 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b0ab5dd-725d-4c8c-9530-311383a99f96-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:53.211578 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.211580 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5tj\" (UniqueName: \"kubernetes.io/projected/2b0ab5dd-725d-4c8c-9530-311383a99f96-kube-api-access-7f5tj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:53.473795 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.473759 4862 generic.go:356] "Generic (PLEG): container finished" podID="2b0ab5dd-725d-4c8c-9530-311383a99f96" containerID="ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad" exitCode=0 Oct 30 19:05:53.474002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.473820 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" Oct 30 19:05:53.474002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.473848 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" event={"ID":"2b0ab5dd-725d-4c8c-9530-311383a99f96","Type":"ContainerDied","Data":"ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad"} Oct 30 19:05:53.474002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.473909 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v" event={"ID":"2b0ab5dd-725d-4c8c-9530-311383a99f96","Type":"ContainerDied","Data":"84d769d563b247fc576872200f3bb18b4eff070215690d7fefc9de1389a86e79"} Oct 30 19:05:53.474002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.473944 4862 scope.go:122] "RemoveContainer" containerID="ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad" Oct 30 19:05:53.488089 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.488065 4862 scope.go:122] "RemoveContainer" containerID="ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad" Oct 30 19:05:53.488440 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:05:53.488419 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad\": container with ID starting with ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad not found: ID does not exist" containerID="ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad" Oct 30 19:05:53.488536 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.488447 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad"} err="failed to get container status \"ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad\": rpc error: code = NotFound desc = could not find container \"ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad\": container with ID starting with ac053fa3a79f26014dd33a3625fafa5e44f8ff2340fc61990abd2756b19a6dad not found: ID does not exist" Oct 30 19:05:53.525043 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.525017 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v"] Oct 30 19:05:53.535287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:53.530639 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-869b55-7f5bdf58fd-q7l8v"] Oct 30 19:05:54.773285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:54.771414 4862 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76597d6855-dxjft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:05:54.773285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:54.771477 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:05:54.818728 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:54.818684 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs"] Oct 30 19:05:54.923981 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:54.923940 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" containerID="cri-o://082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f" gracePeriod=30 Oct 30 19:05:55.123287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.122170 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0ab5dd-725d-4c8c-9530-311383a99f96" path="/var/lib/kubelet/pods/2b0ab5dd-725d-4c8c-9530-311383a99f96/volumes" Oct 30 19:05:55.833858 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.833828 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv"] Oct 30 19:05:55.834497 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.834481 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8503581d-12f9-4af9-a9eb-5bc89d5bde5a" containerName="revoke-keys" Oct 30 19:05:55.834549 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.834498 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8503581d-12f9-4af9-a9eb-5bc89d5bde5a" containerName="revoke-keys" Oct 30 19:05:55.834549 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.834514 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2b0ab5dd-725d-4c8c-9530-311383a99f96" containerName="payload-processing" Oct 30 19:05:55.834549 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.834519 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0ab5dd-725d-4c8c-9530-311383a99f96" containerName="payload-processing" Oct 30 19:05:55.834655 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.834620 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8503581d-12f9-4af9-a9eb-5bc89d5bde5a" containerName="revoke-keys" Oct 30 19:05:55.834655 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.834629 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0ab5dd-725d-4c8c-9530-311383a99f96" containerName="payload-processing" Oct 30 19:05:55.838921 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.838905 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:55.856445 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.856419 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv"] Oct 30 19:05:55.861981 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.861944 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:05:55.861981 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.861953 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:05:55.911075 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.911047 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:55.911324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.911300 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbbv\" (UniqueName: \"kubernetes.io/projected/b2f1a255-4b55-4228-9fae-bcd33833a4b3-kube-api-access-fsbbv\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:55.911407 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:55.911394 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:56.012868 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.012838 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:56.013016 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.013002 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbbv\" (UniqueName: \"kubernetes.io/projected/b2f1a255-4b55-4228-9fae-bcd33833a4b3-kube-api-access-fsbbv\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:56.013070 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.013050 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:56.013482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.013456 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:56.015439 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.015417 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:56.031127 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.031104 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbbv\" (UniqueName: \"kubernetes.io/projected/b2f1a255-4b55-4228-9fae-bcd33833a4b3-kube-api-access-fsbbv\") pod \"maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:56.047675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.047654 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6"] Oct 30 19:05:56.083578 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.083553 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg"] Oct 30 19:05:56.152604 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.152525 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:56.477635 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.477609 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv"] Oct 30 19:05:56.479424 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:05:56.479397 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f1a255_4b55_4228_9fae_bcd33833a4b3.slice/crio-ab2bfe0e6d78b10bd993b38a91332e7b346df76d36a7719e39974cf6c00b9993 WatchSource:0}: Error finding container ab2bfe0e6d78b10bd993b38a91332e7b346df76d36a7719e39974cf6c00b9993: Status 404 returned error can't find the container with id ab2bfe0e6d78b10bd993b38a91332e7b346df76d36a7719e39974cf6c00b9993 Oct 30 19:05:56.557182 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:56.557140 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" event={"ID":"b2f1a255-4b55-4228-9fae-bcd33833a4b3","Type":"ContainerStarted","Data":"ab2bfe0e6d78b10bd993b38a91332e7b346df76d36a7719e39974cf6c00b9993"} Oct 30 19:05:57.568430 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:57.568398 4862 generic.go:356] "Generic (PLEG): container finished" podID="b2f1a255-4b55-4228-9fae-bcd33833a4b3" containerID="9a5a83355fd406dce1f196bb14b4be6fcb160328a70e2c6f53feee4b19a64f5d" exitCode=0 Oct 30 19:05:57.568811 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:57.568463 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" event={"ID":"b2f1a255-4b55-4228-9fae-bcd33833a4b3","Type":"ContainerDied","Data":"9a5a83355fd406dce1f196bb14b4be6fcb160328a70e2c6f53feee4b19a64f5d"} Oct 30 19:05:58.190912 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.190875 4862 patch_prober.go:29] interesting pod/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:05:58.191104 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.190933 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:05:58.756802 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.756775 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:58.860463 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.860393 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-xks-ca\") pod \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " Oct 30 19:05:58.860463 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.860454 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-openshift-ca\") pod \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " Oct 30 19:05:58.860646 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.860509 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f1a255-4b55-4228-9fae-bcd33833a4b3-kube-api-access-fsbbv\" (UniqueName: \"kubernetes.io/projected/b2f1a255-4b55-4228-9fae-bcd33833a4b3-kube-api-access-fsbbv\") pod \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\" (UID: \"b2f1a255-4b55-4228-9fae-bcd33833a4b3\") " Oct 30 19:05:58.860848 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.860815 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-openshift-ca" pod "b2f1a255-4b55-4228-9fae-bcd33833a4b3" (UID: "b2f1a255-4b55-4228-9fae-bcd33833a4b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:58.861159 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.861137 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:58.862799 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.862776 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-xks-ca" pod "b2f1a255-4b55-4228-9fae-bcd33833a4b3" (UID: "b2f1a255-4b55-4228-9fae-bcd33833a4b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:58.862799 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.862781 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f1a255-4b55-4228-9fae-bcd33833a4b3-kube-api-access-fsbbv" pod "b2f1a255-4b55-4228-9fae-bcd33833a4b3" (UID: "b2f1a255-4b55-4228-9fae-bcd33833a4b3"). InnerVolumeSpecName "kube-api-access-fsbbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:58.962554 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.962528 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbbv\" (UniqueName: \"kubernetes.io/projected/b2f1a255-4b55-4228-9fae-bcd33833a4b3-kube-api-access-fsbbv\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:58.962673 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:58.962557 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f1a255-4b55-4228-9fae-bcd33833a4b3-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:05:59.583678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:59.583652 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" Oct 30 19:05:59.583678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:59.583673 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv" event={"ID":"b2f1a255-4b55-4228-9fae-bcd33833a4b3","Type":"ContainerDied","Data":"ab2bfe0e6d78b10bd993b38a91332e7b346df76d36a7719e39974cf6c00b9993"} Oct 30 19:05:59.583892 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:05:59.583723 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab2bfe0e6d78b10bd993b38a91332e7b346df76d36a7719e39974cf6c00b9993" Oct 30 19:06:00.319526 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.319468 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 19:06:00.385960 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.385929 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-certs\") pod \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " Oct 30 19:06:00.387229 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.387200 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-credential-socket\") pod \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " Oct 30 19:06:00.387525 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.387505 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-data\") pod \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " Oct 30 19:06:00.387703 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.387688 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-podinfo\") pod \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " Oct 30 19:06:00.387835 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.387821 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-kube-api-access-kbdz5\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-kube-api-access-kbdz5\") pod \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " Oct 30 19:06:00.387955 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.387943 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-token\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-token\") pod \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " Oct 30 19:06:00.388107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.388093 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-envoy\") pod \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " Oct 30 19:06:00.388237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.388223 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-socket\") pod \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " Oct 30 19:06:00.388446 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.388426 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istiod-ca-cert\") pod \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\" (UID: \"0c42fe79-10c6-4985-8d9e-acdd579ebcca\") " Oct 30 19:06:00.391778 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.391753 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istiod-ca-cert" pod "0c42fe79-10c6-4985-8d9e-acdd579ebcca" (UID: "0c42fe79-10c6-4985-8d9e-acdd579ebcca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:00.393247 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.392509 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-socket" pod "0c42fe79-10c6-4985-8d9e-acdd579ebcca" (UID: "0c42fe79-10c6-4985-8d9e-acdd579ebcca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:00.393247 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.392732 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-certs" pod "0c42fe79-10c6-4985-8d9e-acdd579ebcca" (UID: "0c42fe79-10c6-4985-8d9e-acdd579ebcca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:00.393247 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.392866 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-credential-socket" pod "0c42fe79-10c6-4985-8d9e-acdd579ebcca" (UID: "0c42fe79-10c6-4985-8d9e-acdd579ebcca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:00.393247 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.393005 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-data" pod "0c42fe79-10c6-4985-8d9e-acdd579ebcca" (UID: "0c42fe79-10c6-4985-8d9e-acdd579ebcca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:00.394864 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.394836 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-envoy" pod "0c42fe79-10c6-4985-8d9e-acdd579ebcca" (UID: "0c42fe79-10c6-4985-8d9e-acdd579ebcca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:00.407860 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.407832 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-kube-api-access-kbdz5" pod "0c42fe79-10c6-4985-8d9e-acdd579ebcca" (UID: "0c42fe79-10c6-4985-8d9e-acdd579ebcca"). InnerVolumeSpecName "kube-api-access-kbdz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:00.408973 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.408947 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-token" pod "0c42fe79-10c6-4985-8d9e-acdd579ebcca" (UID: "0c42fe79-10c6-4985-8d9e-acdd579ebcca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:00.417397 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.417368 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-podinfo" pod "0c42fe79-10c6-4985-8d9e-acdd579ebcca" (UID: "0c42fe79-10c6-4985-8d9e-acdd579ebcca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:06:00.492955 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.492922 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:00.492955 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.492947 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:00.492955 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.492958 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:00.493241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.492966 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:00.493241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.492976 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbdz5\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-kube-api-access-kbdz5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:00.493241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.492984 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:00.493241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.492992 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:00.493241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.493000 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c42fe79-10c6-4985-8d9e-acdd579ebcca-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:00.493241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.493008 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c42fe79-10c6-4985-8d9e-acdd579ebcca-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:00.592341 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.592215 4862 generic.go:356] "Generic (PLEG): container finished" podID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerID="082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f" exitCode=0 Oct 30 19:06:00.592341 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.592296 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" event={"ID":"0c42fe79-10c6-4985-8d9e-acdd579ebcca","Type":"ContainerDied","Data":"082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f"} Oct 30 19:06:00.592341 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.592310 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" Oct 30 19:06:00.592594 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.592372 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs" event={"ID":"0c42fe79-10c6-4985-8d9e-acdd579ebcca","Type":"ContainerDied","Data":"7b8076362fa3f35dd0497f299eebbf75b3255765433e9bce22ac2587536a3b75"} Oct 30 19:06:00.592594 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.592395 4862 scope.go:122] "RemoveContainer" containerID="082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f" Oct 30 19:06:00.603972 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.603956 4862 scope.go:122] "RemoveContainer" containerID="496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e" Oct 30 19:06:00.613952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.613933 4862 scope.go:122] "RemoveContainer" containerID="082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f" Oct 30 19:06:00.614241 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:00.614219 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f\": container with ID starting with 082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f not found: ID does not exist" containerID="082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f" Oct 30 19:06:00.614368 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.614256 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f"} err="failed to get container status \"082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f\": rpc error: code = NotFound desc = could not find container \"082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f\": container with ID starting with 082213009ca1686b3ce2c6672961e7469a7ac789492aa610fb0efd1d68bf0e4f not found: ID does not exist" Oct 30 19:06:00.614368 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.614321 4862 scope.go:122] "RemoveContainer" containerID="496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e" Oct 30 19:06:00.614611 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:00.614582 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e\": container with ID starting with 496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e not found: ID does not exist" containerID="496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e" Oct 30 19:06:00.614659 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.614619 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e"} err="failed to get container status \"496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e\": rpc error: code = NotFound desc = could not find container \"496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e\": container with ID starting with 496ce8f3121b9099a44c0f95d3df2613ce56461b4e5f099dbdb5ac628ad7d70e not found: ID does not exist" Oct 30 19:06:00.642806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.642782 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs"] Oct 30 19:06:00.645947 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:00.645928 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2820d9-openshift-default-7b898bbd45-f22vs"] Oct 30 19:06:01.048732 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.048703 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" podUID="af37f402-330d-45e5-8de7-1e6594732b37" containerName="payload-processing" containerID="cri-o://83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe" gracePeriod=25 Oct 30 19:06:01.063495 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.063468 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" path="/var/lib/kubelet/pods/0c42fe79-10c6-4985-8d9e-acdd579ebcca/volumes" Oct 30 19:06:01.084824 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.084799 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" podUID="1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" containerName="payload-pre-processing" containerID="cri-o://d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86" gracePeriod=25 Oct 30 19:06:01.324394 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.324366 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 19:06:01.406280 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.406236 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af37f402-330d-45e5-8de7-1e6594732b37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37f402-330d-45e5-8de7-1e6594732b37-plugins-config-volume\") pod \"af37f402-330d-45e5-8de7-1e6594732b37\" (UID: \"af37f402-330d-45e5-8de7-1e6594732b37\") " Oct 30 19:06:01.406469 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.406373 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af37f402-330d-45e5-8de7-1e6594732b37-kube-api-access-slfdm\" (UniqueName: \"kubernetes.io/projected/af37f402-330d-45e5-8de7-1e6594732b37-kube-api-access-slfdm\") pod \"af37f402-330d-45e5-8de7-1e6594732b37\" (UID: \"af37f402-330d-45e5-8de7-1e6594732b37\") " Oct 30 19:06:01.406603 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.406579 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af37f402-330d-45e5-8de7-1e6594732b37-plugins-config-volume" pod "af37f402-330d-45e5-8de7-1e6594732b37" (UID: "af37f402-330d-45e5-8de7-1e6594732b37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:01.407001 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.406977 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37f402-330d-45e5-8de7-1e6594732b37-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:01.408621 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.408599 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af37f402-330d-45e5-8de7-1e6594732b37-kube-api-access-slfdm" pod "af37f402-330d-45e5-8de7-1e6594732b37" (UID: "af37f402-330d-45e5-8de7-1e6594732b37"). InnerVolumeSpecName "kube-api-access-slfdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:01.508564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.508523 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfdm\" (UniqueName: \"kubernetes.io/projected/af37f402-330d-45e5-8de7-1e6594732b37-kube-api-access-slfdm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:01.601615 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.601530 4862 generic.go:356] "Generic (PLEG): container finished" podID="af37f402-330d-45e5-8de7-1e6594732b37" containerID="83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe" exitCode=0 Oct 30 19:06:01.601615 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.601572 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" Oct 30 19:06:01.601851 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.601615 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" event={"ID":"af37f402-330d-45e5-8de7-1e6594732b37","Type":"ContainerDied","Data":"83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe"} Oct 30 19:06:01.601851 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.601672 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6" event={"ID":"af37f402-330d-45e5-8de7-1e6594732b37","Type":"ContainerDied","Data":"efb6ed46a0e14523d28cc655aab07b3be922ce27f656066cc54cf7dc4ba12b38"} Oct 30 19:06:01.601851 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.601702 4862 scope.go:122] "RemoveContainer" containerID="83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe" Oct 30 19:06:01.613130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.613112 4862 scope.go:122] "RemoveContainer" containerID="83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe" Oct 30 19:06:01.613443 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:01.613424 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe\": container with ID starting with 83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe not found: ID does not exist" containerID="83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe" Oct 30 19:06:01.613504 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.613452 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe"} err="failed to get container status \"83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe\": rpc error: code = NotFound desc = could not find container \"83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe\": container with ID starting with 83d2ef6b9b9cf4a788cf940cb5ef848b0734f0c6f282d3c10d5d660aae896abe not found: ID does not exist" Oct 30 19:06:01.648643 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.648611 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6"] Oct 30 19:06:01.658732 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:01.658710 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-daf3ba-766f4d7b84-bnhf6"] Oct 30 19:06:03.062649 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:03.062622 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af37f402-330d-45e5-8de7-1e6594732b37" path="/var/lib/kubelet/pods/af37f402-330d-45e5-8de7-1e6594732b37/volumes" Oct 30 19:06:05.784042 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:05.784001 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" podUID="1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:06:06.306731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:06.306700 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt"] Oct 30 19:06:06.342286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:06.342233 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6"] Oct 30 19:06:07.636977 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:07.636949 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467"] Oct 30 19:06:07.645991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:07.645968 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-daf3ba-t2467"] Oct 30 19:06:09.063383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:09.063356 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8503581d-12f9-4af9-a9eb-5bc89d5bde5a" path="/var/lib/kubelet/pods/8503581d-12f9-4af9-a9eb-5bc89d5bde5a/volumes" Oct 30 19:06:11.307434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.307379 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" podUID="3425a122-a8c3-4339-9665-a4df3b7da91c" containerName="payload-processing" containerID="cri-o://d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072" gracePeriod=25 Oct 30 19:06:11.342582 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.342533 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" podUID="ff6ec8bb-742e-46e1-80b1-01982cda2ea3" containerName="payload-pre-processing" containerID="cri-o://751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8" gracePeriod=25 Oct 30 19:06:11.614492 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.614463 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 19:06:11.651691 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.651669 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3425a122-a8c3-4339-9665-a4df3b7da91c-kube-api-access-9tncn\" (UniqueName: \"kubernetes.io/projected/3425a122-a8c3-4339-9665-a4df3b7da91c-kube-api-access-9tncn\") pod \"3425a122-a8c3-4339-9665-a4df3b7da91c\" (UID: \"3425a122-a8c3-4339-9665-a4df3b7da91c\") " Oct 30 19:06:11.651834 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.651822 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3425a122-a8c3-4339-9665-a4df3b7da91c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3425a122-a8c3-4339-9665-a4df3b7da91c-plugins-config-volume\") pod \"3425a122-a8c3-4339-9665-a4df3b7da91c\" (UID: \"3425a122-a8c3-4339-9665-a4df3b7da91c\") " Oct 30 19:06:11.652122 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.652098 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3425a122-a8c3-4339-9665-a4df3b7da91c-plugins-config-volume" pod "3425a122-a8c3-4339-9665-a4df3b7da91c" (UID: "3425a122-a8c3-4339-9665-a4df3b7da91c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:11.652401 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.652240 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3425a122-a8c3-4339-9665-a4df3b7da91c-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:11.653898 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.653873 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3425a122-a8c3-4339-9665-a4df3b7da91c-kube-api-access-9tncn" pod "3425a122-a8c3-4339-9665-a4df3b7da91c" (UID: "3425a122-a8c3-4339-9665-a4df3b7da91c"). InnerVolumeSpecName "kube-api-access-9tncn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:11.681986 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.681940 4862 generic.go:356] "Generic (PLEG): container finished" podID="3425a122-a8c3-4339-9665-a4df3b7da91c" containerID="d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072" exitCode=0 Oct 30 19:06:11.682134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.681991 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" Oct 30 19:06:11.682134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.682016 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" event={"ID":"3425a122-a8c3-4339-9665-a4df3b7da91c","Type":"ContainerDied","Data":"d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072"} Oct 30 19:06:11.682134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.682081 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt" event={"ID":"3425a122-a8c3-4339-9665-a4df3b7da91c","Type":"ContainerDied","Data":"e31ced28b8b96600275c3eb11ae97eb480ac2152230607de087efcb65dc9155c"} Oct 30 19:06:11.682134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.682105 4862 scope.go:122] "RemoveContainer" containerID="d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072" Oct 30 19:06:11.694801 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.694785 4862 scope.go:122] "RemoveContainer" containerID="d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072" Oct 30 19:06:11.695068 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:11.695037 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072\": container with ID starting with d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072 not found: ID does not exist" containerID="d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072" Oct 30 19:06:11.695188 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.695078 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072"} err="failed to get container status \"d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072\": rpc error: code = NotFound desc = could not find container \"d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072\": container with ID starting with d22f85518d811eb7f8ece782f91bfa12ce55cd1ff49f1f05724a06e0043c8072 not found: ID does not exist" Oct 30 19:06:11.729414 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.729393 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt"] Oct 30 19:06:11.732844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.732823 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9f6b7c-59d7fd9cb-fh4tt"] Oct 30 19:06:11.753248 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:11.753225 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tncn\" (UniqueName: \"kubernetes.io/projected/3425a122-a8c3-4339-9665-a4df3b7da91c-kube-api-access-9tncn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:13.063545 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:13.063511 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3425a122-a8c3-4339-9665-a4df3b7da91c" path="/var/lib/kubelet/pods/3425a122-a8c3-4339-9665-a4df3b7da91c/volumes" Oct 30 19:06:14.777488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:14.777444 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz"] Oct 30 19:06:14.778134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:14.778100 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" containerID="cri-o://802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2" gracePeriod=30 Oct 30 19:06:15.396151 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396121 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp"] Oct 30 19:06:15.396781 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396762 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" Oct 30 19:06:15.396781 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396780 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" Oct 30 19:06:15.396946 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396816 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b2f1a255-4b55-4228-9fae-bcd33833a4b3" containerName="revoke-keys" Oct 30 19:06:15.396946 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396821 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f1a255-4b55-4228-9fae-bcd33833a4b3" containerName="revoke-keys" Oct 30 19:06:15.396946 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396838 4862 cpu_manager.go:414] "containerMap: removing container" podUID="af37f402-330d-45e5-8de7-1e6594732b37" containerName="payload-processing" Oct 30 19:06:15.396946 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396843 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af37f402-330d-45e5-8de7-1e6594732b37" containerName="payload-processing" Oct 30 19:06:15.396946 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396853 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3425a122-a8c3-4339-9665-a4df3b7da91c" containerName="payload-processing" Oct 30 19:06:15.396946 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396863 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3425a122-a8c3-4339-9665-a4df3b7da91c" containerName="payload-processing" Oct 30 19:06:15.397237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396972 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" Oct 30 19:06:15.397237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396981 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" Oct 30 19:06:15.397237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.396992 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f1a255-4b55-4228-9fae-bcd33833a4b3" containerName="revoke-keys" Oct 30 19:06:15.397237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.397000 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="af37f402-330d-45e5-8de7-1e6594732b37" containerName="payload-processing" Oct 30 19:06:15.397237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.397006 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3425a122-a8c3-4339-9665-a4df3b7da91c" containerName="payload-processing" Oct 30 19:06:15.401161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.401142 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.416140 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.416115 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c095fa\"/\"sub-test-model-c095fa-kserve-self-signed-certs\"" Oct 30 19:06:15.423241 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.423216 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp"] Oct 30 19:06:15.429498 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.429475 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c095fa\"/\"default-dockercfg-27k42\"" Oct 30 19:06:15.483694 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.483666 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c095fa\"/\"kube-root-ca.crt\"" Oct 30 19:06:15.483902 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.483703 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c095fa\"/\"openshift-service-ca.crt\"" Oct 30 19:06:15.508528 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.508488 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-home\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.508677 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.508572 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbc48\" (UniqueName: \"kubernetes.io/projected/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-kube-api-access-tbc48\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.508750 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.508713 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tls-certs\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.508820 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.508752 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-dshm\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.508874 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.508843 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tmp-dir\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.508929 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.508912 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-model-cache\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.610378 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.610345 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-model-cache\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.610587 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.610571 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-home\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.610658 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.610629 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbc48\" (UniqueName: \"kubernetes.io/projected/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-kube-api-access-tbc48\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.610781 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.610704 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tls-certs\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.611107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.610847 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-model-cache\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.611107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.610954 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-home\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.611107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.610980 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-dshm\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.611107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.611056 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tmp-dir\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.612381 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.611467 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tmp-dir\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.613789 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.613764 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-dshm\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.642288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.641932 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tls-certs\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.652469 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.646639 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbc48\" (UniqueName: \"kubernetes.io/projected/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-kube-api-access-tbc48\") pod \"sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.724901 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.724668 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:15.786253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:15.786214 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" podUID="1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:06:16.075894 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:16.075872 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp"] Oct 30 19:06:16.078309 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:06:16.078256 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef7feae8_dfc9_4e75_b95e_e1a29a5f1729.slice/crio-e95d7377c35e4971cc576bbbbf2a757c5dffe683f4e740b1ce83d71721a7d7f4 WatchSource:0}: Error finding container e95d7377c35e4971cc576bbbbf2a757c5dffe683f4e740b1ce83d71721a7d7f4: Status 404 returned error can't find the container with id e95d7377c35e4971cc576bbbbf2a757c5dffe683f4e740b1ce83d71721a7d7f4 Oct 30 19:06:16.770235 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:16.770196 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" event={"ID":"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729","Type":"ContainerStarted","Data":"7e3a330c6828d224d51c84d1907bb67f15697dd07ee7c1d5b997d6b5ed7d67db"} Oct 30 19:06:16.770455 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:16.770243 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" event={"ID":"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729","Type":"ContainerStarted","Data":"e95d7377c35e4971cc576bbbbf2a757c5dffe683f4e740b1ce83d71721a7d7f4"} Oct 30 19:06:16.811077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:16.811030 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" podStartSLOduration=1.8110165139999999 podStartE2EDuration="1.811016514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:16.809436084 +0000 UTC m=+4075.858480993" watchObservedRunningTime="2026-10-30 19:06:16.811016514 +0000 UTC m=+4075.860061426" Oct 30 19:06:17.729907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:17.729873 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:18.118933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:18.118895 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" podUID="ff6ec8bb-742e-46e1-80b1-01982cda2ea3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:06:19.871419 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:19.871386 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:06:20.053235 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.053212 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 19:06:20.186887 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.186856 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-kube-api-access-lgzgg\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-kube-api-access-lgzgg\") pod \"a32dd5fc-27df-453f-b026-3811e7fac2f1\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " Oct 30 19:06:20.187049 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.186898 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-podinfo\") pod \"a32dd5fc-27df-453f-b026-3811e7fac2f1\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " Oct 30 19:06:20.187049 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.186938 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-socket\") pod \"a32dd5fc-27df-453f-b026-3811e7fac2f1\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " Oct 30 19:06:20.187049 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187047 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32dd5fc-27df-453f-b026-3811e7fac2f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32dd5fc-27df-453f-b026-3811e7fac2f1-istiod-ca-cert\") pod \"a32dd5fc-27df-453f-b026-3811e7fac2f1\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " Oct 30 19:06:20.187206 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187081 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-credential-socket\") pod \"a32dd5fc-27df-453f-b026-3811e7fac2f1\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " Oct 30 19:06:20.187206 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187150 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-data\") pod \"a32dd5fc-27df-453f-b026-3811e7fac2f1\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " Oct 30 19:06:20.187206 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187187 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-certs\") pod \"a32dd5fc-27df-453f-b026-3811e7fac2f1\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " Oct 30 19:06:20.187461 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187282 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-token\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-token\") pod \"a32dd5fc-27df-453f-b026-3811e7fac2f1\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " Oct 30 19:06:20.187461 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187316 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-envoy\") pod \"a32dd5fc-27df-453f-b026-3811e7fac2f1\" (UID: \"a32dd5fc-27df-453f-b026-3811e7fac2f1\") " Oct 30 19:06:20.187461 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187350 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-socket" pod "a32dd5fc-27df-453f-b026-3811e7fac2f1" (UID: "a32dd5fc-27df-453f-b026-3811e7fac2f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:20.187461 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187395 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-credential-socket" pod "a32dd5fc-27df-453f-b026-3811e7fac2f1" (UID: "a32dd5fc-27df-453f-b026-3811e7fac2f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:20.187699 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187478 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32dd5fc-27df-453f-b026-3811e7fac2f1-istiod-ca-cert" pod "a32dd5fc-27df-453f-b026-3811e7fac2f1" (UID: "a32dd5fc-27df-453f-b026-3811e7fac2f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:20.187699 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187516 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-certs" pod "a32dd5fc-27df-453f-b026-3811e7fac2f1" (UID: "a32dd5fc-27df-453f-b026-3811e7fac2f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:20.187823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187767 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-data" pod "a32dd5fc-27df-453f-b026-3811e7fac2f1" (UID: "a32dd5fc-27df-453f-b026-3811e7fac2f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:20.187972 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187946 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:20.187972 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187972 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:20.188156 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.187986 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:20.188156 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.188000 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32dd5fc-27df-453f-b026-3811e7fac2f1-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:20.188156 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.188012 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:20.189562 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.189536 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-kube-api-access-lgzgg" pod "a32dd5fc-27df-453f-b026-3811e7fac2f1" (UID: "a32dd5fc-27df-453f-b026-3811e7fac2f1"). InnerVolumeSpecName "kube-api-access-lgzgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:20.189989 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.189956 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-envoy" pod "a32dd5fc-27df-453f-b026-3811e7fac2f1" (UID: "a32dd5fc-27df-453f-b026-3811e7fac2f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:20.190141 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.190117 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-podinfo" pod "a32dd5fc-27df-453f-b026-3811e7fac2f1" (UID: "a32dd5fc-27df-453f-b026-3811e7fac2f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:06:20.190238 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.190167 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-token" pod "a32dd5fc-27df-453f-b026-3811e7fac2f1" (UID: "a32dd5fc-27df-453f-b026-3811e7fac2f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:20.289560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.289532 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:20.289560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.289554 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:20.289560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.289564 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgzgg\" (UniqueName: \"kubernetes.io/projected/a32dd5fc-27df-453f-b026-3811e7fac2f1-kube-api-access-lgzgg\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:20.289761 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.289573 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32dd5fc-27df-453f-b026-3811e7fac2f1-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:20.822222 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.822145 4862 generic.go:356] "Generic (PLEG): container finished" podID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerID="802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2" exitCode=0 Oct 30 19:06:20.822222 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.822208 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" Oct 30 19:06:20.822421 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.822236 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" event={"ID":"a32dd5fc-27df-453f-b026-3811e7fac2f1","Type":"ContainerDied","Data":"802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2"} Oct 30 19:06:20.822421 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.822341 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz" event={"ID":"a32dd5fc-27df-453f-b026-3811e7fac2f1","Type":"ContainerDied","Data":"672c00ca881f6262191c6a142ffcebd3bd456a0428d1434b87e8103954c93d6a"} Oct 30 19:06:20.822421 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.822365 4862 scope.go:122] "RemoveContainer" containerID="802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2" Oct 30 19:06:20.834275 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.834247 4862 scope.go:122] "RemoveContainer" containerID="860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1" Oct 30 19:06:20.843882 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.843865 4862 scope.go:122] "RemoveContainer" containerID="802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2" Oct 30 19:06:20.844122 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:20.844102 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2\": container with ID starting with 802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2 not found: ID does not exist" containerID="802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2" Oct 30 19:06:20.844178 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.844130 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2"} err="failed to get container status \"802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2\": rpc error: code = NotFound desc = could not find container \"802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2\": container with ID starting with 802e2647fbba8444dc87946ec609363d2044748d3e8dabf4341c5ddf2c01b1e2 not found: ID does not exist" Oct 30 19:06:20.844178 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.844146 4862 scope.go:122] "RemoveContainer" containerID="860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1" Oct 30 19:06:20.844353 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:20.844336 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1\": container with ID starting with 860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1 not found: ID does not exist" containerID="860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1" Oct 30 19:06:20.844402 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.844357 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1"} err="failed to get container status \"860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1\": rpc error: code = NotFound desc = could not find container \"860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1\": container with ID starting with 860c67ad120c34ee9914dc898c18febd003d94dbffc0dae50e1dc1bdb311cff1 not found: ID does not exist" Oct 30 19:06:20.885833 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.885800 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz"] Oct 30 19:06:20.890288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:20.890253 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-daf3ba-openshift-default-68fbcb9679-hlwgz"] Oct 30 19:06:21.063389 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.063362 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" path="/var/lib/kubelet/pods/a32dd5fc-27df-453f-b026-3811e7fac2f1/volumes" Oct 30 19:06:21.356627 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.356573 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 19:06:21.504734 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.504696 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-kube-api-access-n6bnq\" (UniqueName: \"kubernetes.io/projected/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-kube-api-access-n6bnq\") pod \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\" (UID: \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\") " Oct 30 19:06:21.504734 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.504807 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-plugins-config-volume\") pod \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\" (UID: \"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1\") " Oct 30 19:06:21.505212 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.505189 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-plugins-config-volume" pod "1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" (UID: "1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:21.505434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.505399 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:21.507402 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.507376 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-kube-api-access-n6bnq" pod "1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" (UID: "1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1"). InnerVolumeSpecName "kube-api-access-n6bnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:21.607057 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.606995 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6bnq\" (UniqueName: \"kubernetes.io/projected/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1-kube-api-access-n6bnq\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:21.831564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.831533 4862 generic.go:356] "Generic (PLEG): container finished" podID="1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" containerID="d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86" exitCode=0 Oct 30 19:06:21.831728 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.831582 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" Oct 30 19:06:21.831728 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.831621 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" event={"ID":"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1","Type":"ContainerDied","Data":"d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86"} Oct 30 19:06:21.831728 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.831676 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg" event={"ID":"1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1","Type":"ContainerDied","Data":"6574b1a075d0013d9bebfd58ea8f60a58d25019fad671b6a533a1eb2282a5423"} Oct 30 19:06:21.831728 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.831702 4862 scope.go:122] "RemoveContainer" containerID="d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86" Oct 30 19:06:21.842788 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.842773 4862 scope.go:122] "RemoveContainer" containerID="d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86" Oct 30 19:06:21.843005 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:21.842987 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86\": container with ID starting with d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86 not found: ID does not exist" containerID="d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86" Oct 30 19:06:21.843053 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.843013 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86"} err="failed to get container status \"d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86\": rpc error: code = NotFound desc = could not find container \"d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86\": container with ID starting with d6fdf67c265dd9b792a38526c5e73774324a9c031665ff59c1ad883a3e95ad86 not found: ID does not exist" Oct 30 19:06:21.878474 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.878425 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg"] Oct 30 19:06:21.888700 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:21.888678 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-daf3ba-7d69f954dc-9rrgg"] Oct 30 19:06:23.063140 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:23.063107 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" path="/var/lib/kubelet/pods/1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1/volumes" Oct 30 19:06:23.654533 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:23.654503 4862 scope.go:122] "RemoveContainer" containerID="def9a9bf51d92f009f6853fa1d86447c565c9e8f978e06409fa2f139b5a224a6" Oct 30 19:06:25.729393 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:25.729357 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:25.770913 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:25.770828 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:25.874962 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:25.874935 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:06:28.119274 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:28.119233 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" podUID="ff6ec8bb-742e-46e1-80b1-01982cda2ea3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:06:30.760483 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:30.760447 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:06:31.699768 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.699738 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 19:06:31.754605 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.754572 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-plugins-config-volume\") pod \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\" (UID: \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\") " Oct 30 19:06:31.757460 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.754778 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-kube-api-access-89lpc\" (UniqueName: \"kubernetes.io/projected/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-kube-api-access-89lpc\") pod \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\" (UID: \"ff6ec8bb-742e-46e1-80b1-01982cda2ea3\") " Oct 30 19:06:31.757460 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.755231 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-plugins-config-volume" pod "ff6ec8bb-742e-46e1-80b1-01982cda2ea3" (UID: "ff6ec8bb-742e-46e1-80b1-01982cda2ea3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:31.758246 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.758217 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-kube-api-access-89lpc" pod "ff6ec8bb-742e-46e1-80b1-01982cda2ea3" (UID: "ff6ec8bb-742e-46e1-80b1-01982cda2ea3"). InnerVolumeSpecName "kube-api-access-89lpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:31.856596 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.856557 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lpc\" (UniqueName: \"kubernetes.io/projected/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-kube-api-access-89lpc\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:31.856596 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.856591 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff6ec8bb-742e-46e1-80b1-01982cda2ea3-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:31.908576 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.908543 4862 generic.go:356] "Generic (PLEG): container finished" podID="ff6ec8bb-742e-46e1-80b1-01982cda2ea3" containerID="751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8" exitCode=0 Oct 30 19:06:31.908770 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.908589 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" Oct 30 19:06:31.908770 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.908621 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" event={"ID":"ff6ec8bb-742e-46e1-80b1-01982cda2ea3","Type":"ContainerDied","Data":"751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8"} Oct 30 19:06:31.908770 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.908673 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6" event={"ID":"ff6ec8bb-742e-46e1-80b1-01982cda2ea3","Type":"ContainerDied","Data":"8ae0653a978fba0c4a4d5bb9fbb734464fbbbc789702d68f4fba7e84970ccfe6"} Oct 30 19:06:31.908770 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.908695 4862 scope.go:122] "RemoveContainer" containerID="751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8" Oct 30 19:06:31.923836 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.923790 4862 scope.go:122] "RemoveContainer" containerID="751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8" Oct 30 19:06:31.925068 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:31.924972 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8\": container with ID starting with 751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8 not found: ID does not exist" containerID="751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8" Oct 30 19:06:31.925068 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.925026 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8"} err="failed to get container status \"751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8\": rpc error: code = NotFound desc = could not find container \"751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8\": container with ID starting with 751f52a79e8f64aea9f6648f184cddff1c6de7f223caa72e5000af830abecad8 not found: ID does not exist" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.954341 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z"] Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955340 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955361 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955383 4862 cpu_manager.go:414] "containerMap: removing container" podUID="1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" containerName="payload-pre-processing" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955391 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" containerName="payload-pre-processing" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955449 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955459 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955499 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ff6ec8bb-742e-46e1-80b1-01982cda2ea3" containerName="payload-pre-processing" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955507 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6ec8bb-742e-46e1-80b1-01982cda2ea3" containerName="payload-pre-processing" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955539 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955547 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c42fe79-10c6-4985-8d9e-acdd579ebcca" containerName="istio-proxy" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955781 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955796 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3cbe16-00bd-4ae8-9e1d-5e5420a014d1" containerName="payload-pre-processing" Oct 30 19:06:31.969823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:31.955817 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6ec8bb-742e-46e1-80b1-01982cda2ea3" containerName="payload-pre-processing" Oct 30 19:06:32.006352 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.001897 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6"] Oct 30 19:06:32.006352 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.001943 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z"] Oct 30 19:06:32.006352 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.002048 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.006352 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.003999 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9f6b7c-549554d7c9-jrdl6"] Oct 30 19:06:32.029490 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.023863 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:06:32.029490 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.024607 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:06:32.180154 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.180071 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0356d9d-v492z\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.180394 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.180228 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0356d9d-v492z\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.180394 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.180285 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2k24\" (UniqueName: \"kubernetes.io/projected/cefea2ca-0c8a-4ade-93f4-35042318fa38-kube-api-access-g2k24\") pod \"maas-api-revoke-sub-b0356d9d-v492z\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.282085 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.282041 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0356d9d-v492z\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.282357 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.282333 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0356d9d-v492z\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.282487 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.282437 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2k24\" (UniqueName: \"kubernetes.io/projected/cefea2ca-0c8a-4ade-93f4-35042318fa38-kube-api-access-g2k24\") pod \"maas-api-revoke-sub-b0356d9d-v492z\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.283003 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.282978 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0356d9d-v492z\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.284699 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.284677 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0356d9d-v492z\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.304071 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.304051 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2k24\" (UniqueName: \"kubernetes.io/projected/cefea2ca-0c8a-4ade-93f4-35042318fa38-kube-api-access-g2k24\") pod \"maas-api-revoke-sub-b0356d9d-v492z\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.367936 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.367885 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:32.685613 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.685522 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z"] Oct 30 19:06:32.691859 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:06:32.691828 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefea2ca_0c8a_4ade_93f4_35042318fa38.slice/crio-5323bae5670ec6336c791441404e9411323bd63a5bd9c5494da3e45d58f4617c WatchSource:0}: Error finding container 5323bae5670ec6336c791441404e9411323bd63a5bd9c5494da3e45d58f4617c: Status 404 returned error can't find the container with id 5323bae5670ec6336c791441404e9411323bd63a5bd9c5494da3e45d58f4617c Oct 30 19:06:32.918325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:32.918285 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" event={"ID":"cefea2ca-0c8a-4ade-93f4-35042318fa38","Type":"ContainerStarted","Data":"5323bae5670ec6336c791441404e9411323bd63a5bd9c5494da3e45d58f4617c"} Oct 30 19:06:33.075587 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:33.075539 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6ec8bb-742e-46e1-80b1-01982cda2ea3" path="/var/lib/kubelet/pods/ff6ec8bb-742e-46e1-80b1-01982cda2ea3/volumes" Oct 30 19:06:33.929677 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:33.929638 4862 generic.go:356] "Generic (PLEG): container finished" podID="cefea2ca-0c8a-4ade-93f4-35042318fa38" containerID="d32fd5e2a66c854a9862792b2f6c285ff5d7e3ba3afc2c0f91742d2781e36d43" exitCode=0 Oct 30 19:06:33.930228 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:33.929733 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" event={"ID":"cefea2ca-0c8a-4ade-93f4-35042318fa38","Type":"ContainerDied","Data":"d32fd5e2a66c854a9862792b2f6c285ff5d7e3ba3afc2c0f91742d2781e36d43"} Oct 30 19:06:35.106976 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.106948 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:35.127129 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.127104 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-openshift-ca\") pod \"cefea2ca-0c8a-4ade-93f4-35042318fa38\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " Oct 30 19:06:35.127292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.127236 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-xks-ca\") pod \"cefea2ca-0c8a-4ade-93f4-35042318fa38\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " Oct 30 19:06:35.127356 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.127306 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefea2ca-0c8a-4ade-93f4-35042318fa38-kube-api-access-g2k24\" (UniqueName: \"kubernetes.io/projected/cefea2ca-0c8a-4ade-93f4-35042318fa38-kube-api-access-g2k24\") pod \"cefea2ca-0c8a-4ade-93f4-35042318fa38\" (UID: \"cefea2ca-0c8a-4ade-93f4-35042318fa38\") " Oct 30 19:06:35.127524 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.127504 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-openshift-ca" pod "cefea2ca-0c8a-4ade-93f4-35042318fa38" (UID: "cefea2ca-0c8a-4ade-93f4-35042318fa38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:35.128119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.128093 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:35.129657 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.129629 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-xks-ca" pod "cefea2ca-0c8a-4ade-93f4-35042318fa38" (UID: "cefea2ca-0c8a-4ade-93f4-35042318fa38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:06:35.129783 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.129703 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefea2ca-0c8a-4ade-93f4-35042318fa38-kube-api-access-g2k24" pod "cefea2ca-0c8a-4ade-93f4-35042318fa38" (UID: "cefea2ca-0c8a-4ade-93f4-35042318fa38"). InnerVolumeSpecName "kube-api-access-g2k24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:35.229140 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.229059 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cefea2ca-0c8a-4ade-93f4-35042318fa38-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:35.229140 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.229089 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2k24\" (UniqueName: \"kubernetes.io/projected/cefea2ca-0c8a-4ade-93f4-35042318fa38-kube-api-access-g2k24\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:06:35.946604 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.946576 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" Oct 30 19:06:35.946788 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.946665 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z" event={"ID":"cefea2ca-0c8a-4ade-93f4-35042318fa38","Type":"ContainerDied","Data":"5323bae5670ec6336c791441404e9411323bd63a5bd9c5494da3e45d58f4617c"} Oct 30 19:06:35.946788 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:35.946726 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5323bae5670ec6336c791441404e9411323bd63a5bd9c5494da3e45d58f4617c" Oct 30 19:06:49.869596 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:06:49.869556 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:06:55.321874 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:55.321841 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv"] Oct 30 19:06:55.331245 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:55.331218 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9f6b7c-wb4lv"] Oct 30 19:06:57.062396 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:06:57.062363 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f1a255-4b55-4228-9fae-bcd33833a4b3" path="/var/lib/kubelet/pods/b2f1a255-4b55-4228-9fae-bcd33833a4b3/volumes" Oct 30 19:07:10.719561 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.719526 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9"] Oct 30 19:07:10.720150 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.720134 4862 cpu_manager.go:414] "containerMap: removing container" podUID="cefea2ca-0c8a-4ade-93f4-35042318fa38" containerName="revoke-keys" Oct 30 19:07:10.720197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.720151 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefea2ca-0c8a-4ade-93f4-35042318fa38" containerName="revoke-keys" Oct 30 19:07:10.720302 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.720291 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefea2ca-0c8a-4ade-93f4-35042318fa38" containerName="revoke-keys" Oct 30 19:07:10.720347 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.720309 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32dd5fc-27df-453f-b026-3811e7fac2f1" containerName="istio-proxy" Oct 30 19:07:10.723960 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.723940 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:10.738148 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.738127 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:07:10.743313 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.743287 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9"] Oct 30 19:07:10.748732 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.748707 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:07:10.848571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.848539 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:10.848728 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.848595 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ptd\" (UniqueName: \"kubernetes.io/projected/3a5304b2-799f-431e-bc86-563c681ebc69-kube-api-access-q9ptd\") pod \"maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:10.848728 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.848693 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:10.949757 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.949726 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:10.949918 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.949792 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ptd\" (UniqueName: \"kubernetes.io/projected/3a5304b2-799f-431e-bc86-563c681ebc69-kube-api-access-q9ptd\") pod \"maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:10.949918 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.949836 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:10.950506 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.950477 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:10.952406 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.952383 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:10.967842 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:10.967821 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ptd\" (UniqueName: \"kubernetes.io/projected/3a5304b2-799f-431e-bc86-563c681ebc69-kube-api-access-q9ptd\") pod \"maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:11.037626 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:11.037572 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:11.337247 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:11.337224 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9"] Oct 30 19:07:11.340195 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:07:11.340162 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5304b2_799f_431e_bc86_563c681ebc69.slice/crio-92f57017831e0aac87b8f6f988fdfba8eacee95c40ae668eb54d0d39cb8d09a8 WatchSource:0}: Error finding container 92f57017831e0aac87b8f6f988fdfba8eacee95c40ae668eb54d0d39cb8d09a8: Status 404 returned error can't find the container with id 92f57017831e0aac87b8f6f988fdfba8eacee95c40ae668eb54d0d39cb8d09a8 Oct 30 19:07:12.239332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:12.239297 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" event={"ID":"3a5304b2-799f-431e-bc86-563c681ebc69","Type":"ContainerStarted","Data":"82087453256bb3371d5132060b66523770680df73c1ee43d2fa828bab8a95eff"} Oct 30 19:07:12.239727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:12.239362 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" event={"ID":"3a5304b2-799f-431e-bc86-563c681ebc69","Type":"ContainerStarted","Data":"92f57017831e0aac87b8f6f988fdfba8eacee95c40ae668eb54d0d39cb8d09a8"} Oct 30 19:07:12.279002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:12.278948 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" podStartSLOduration=2.2789298049999998 podStartE2EDuration="2.278929805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:12.277465691 +0000 UTC m=+4131.326510607" watchObservedRunningTime="2026-10-30 19:07:12.278929805 +0000 UTC m=+4131.327974720" Oct 30 19:07:13.250393 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:13.250351 4862 generic.go:356] "Generic (PLEG): container finished" podID="3a5304b2-799f-431e-bc86-563c681ebc69" containerID="82087453256bb3371d5132060b66523770680df73c1ee43d2fa828bab8a95eff" exitCode=0 Oct 30 19:07:13.250912 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:13.250402 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" event={"ID":"3a5304b2-799f-431e-bc86-563c681ebc69","Type":"ContainerDied","Data":"82087453256bb3371d5132060b66523770680df73c1ee43d2fa828bab8a95eff"} Oct 30 19:07:14.811431 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:14.811402 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:14.903289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:14.891728 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5304b2-799f-431e-bc86-563c681ebc69-kube-api-access-q9ptd\" (UniqueName: \"kubernetes.io/projected/3a5304b2-799f-431e-bc86-563c681ebc69-kube-api-access-q9ptd\") pod \"3a5304b2-799f-431e-bc86-563c681ebc69\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " Oct 30 19:07:14.903289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:14.891875 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-xks-ca\") pod \"3a5304b2-799f-431e-bc86-563c681ebc69\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " Oct 30 19:07:14.903289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:14.892067 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-openshift-ca\") pod \"3a5304b2-799f-431e-bc86-563c681ebc69\" (UID: \"3a5304b2-799f-431e-bc86-563c681ebc69\") " Oct 30 19:07:14.913285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:14.904634 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-openshift-ca" pod "3a5304b2-799f-431e-bc86-563c681ebc69" (UID: "3a5304b2-799f-431e-bc86-563c681ebc69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:14.913285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:14.907805 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:14.923953 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:14.922488 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-xks-ca" pod "3a5304b2-799f-431e-bc86-563c681ebc69" (UID: "3a5304b2-799f-431e-bc86-563c681ebc69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:14.924426 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:14.924388 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5304b2-799f-431e-bc86-563c681ebc69-kube-api-access-q9ptd" pod "3a5304b2-799f-431e-bc86-563c681ebc69" (UID: "3a5304b2-799f-431e-bc86-563c681ebc69"). InnerVolumeSpecName "kube-api-access-q9ptd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:15.009939 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:15.009842 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9ptd\" (UniqueName: \"kubernetes.io/projected/3a5304b2-799f-431e-bc86-563c681ebc69-kube-api-access-q9ptd\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:15.009939 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:15.009888 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a5304b2-799f-431e-bc86-563c681ebc69-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:15.292806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:15.292730 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" Oct 30 19:07:15.292806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:15.292769 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9" event={"ID":"3a5304b2-799f-431e-bc86-563c681ebc69","Type":"ContainerDied","Data":"92f57017831e0aac87b8f6f988fdfba8eacee95c40ae668eb54d0d39cb8d09a8"} Oct 30 19:07:15.293000 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:15.292814 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92f57017831e0aac87b8f6f988fdfba8eacee95c40ae668eb54d0d39cb8d09a8" Oct 30 19:07:18.470039 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.470002 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8"] Oct 30 19:07:18.470807 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.470783 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3a5304b2-799f-431e-bc86-563c681ebc69" containerName="revoke-keys" Oct 30 19:07:18.470915 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.470809 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5304b2-799f-431e-bc86-563c681ebc69" containerName="revoke-keys" Oct 30 19:07:18.471035 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.471020 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5304b2-799f-431e-bc86-563c681ebc69" containerName="revoke-keys" Oct 30 19:07:18.474749 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.474728 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.494562 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.494532 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8"] Oct 30 19:07:18.497771 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.497748 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:07:18.497915 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.497899 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:07:18.552348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.552320 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.552495 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.552388 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghtc\" (UniqueName: \"kubernetes.io/projected/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-kube-api-access-kghtc\") pod \"maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.552552 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.552529 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.654187 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.654153 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.654333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.654219 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghtc\" (UniqueName: \"kubernetes.io/projected/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-kube-api-access-kghtc\") pod \"maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.654333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.654329 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.654916 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.654890 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.656653 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.656634 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.671227 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.671208 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghtc\" (UniqueName: \"kubernetes.io/projected/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-kube-api-access-kghtc\") pod \"maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:18.799354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:18.799331 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:19.105821 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:19.105789 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8"] Oct 30 19:07:19.107385 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:07:19.107354 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2d6afc_c986_4ec7_85fb_1a8762bb4943.slice/crio-34010bcf8ba191075caa5d5b048c57bb1d2b1ae3519ce6615198ec336c005580 WatchSource:0}: Error finding container 34010bcf8ba191075caa5d5b048c57bb1d2b1ae3519ce6615198ec336c005580: Status 404 returned error can't find the container with id 34010bcf8ba191075caa5d5b048c57bb1d2b1ae3519ce6615198ec336c005580 Oct 30 19:07:19.325683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:19.325650 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" event={"ID":"4a2d6afc-c986-4ec7-85fb-1a8762bb4943","Type":"ContainerStarted","Data":"34010bcf8ba191075caa5d5b048c57bb1d2b1ae3519ce6615198ec336c005580"} Oct 30 19:07:19.873139 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:07:19.873089 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:07:20.334536 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:20.334508 4862 generic.go:356] "Generic (PLEG): container finished" podID="4a2d6afc-c986-4ec7-85fb-1a8762bb4943" containerID="a55a1ad183beae5d53aa5a097952b65931e9a346a34d5aedd21fe8ad23db92cf" exitCode=0 Oct 30 19:07:20.334713 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:20.334603 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" event={"ID":"4a2d6afc-c986-4ec7-85fb-1a8762bb4943","Type":"ContainerDied","Data":"a55a1ad183beae5d53aa5a097952b65931e9a346a34d5aedd21fe8ad23db92cf"} Oct 30 19:07:21.549464 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.549440 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:21.589122 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.589083 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-kube-api-access-kghtc\" (UniqueName: \"kubernetes.io/projected/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-kube-api-access-kghtc\") pod \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " Oct 30 19:07:21.589394 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.589368 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-xks-ca\") pod \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " Oct 30 19:07:21.589509 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.589471 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-openshift-ca\") pod \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\" (UID: \"4a2d6afc-c986-4ec7-85fb-1a8762bb4943\") " Oct 30 19:07:21.590865 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.590836 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-openshift-ca" pod "4a2d6afc-c986-4ec7-85fb-1a8762bb4943" (UID: "4a2d6afc-c986-4ec7-85fb-1a8762bb4943"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:21.593574 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.593519 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-xks-ca" pod "4a2d6afc-c986-4ec7-85fb-1a8762bb4943" (UID: "4a2d6afc-c986-4ec7-85fb-1a8762bb4943"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:21.594462 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.594436 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-kube-api-access-kghtc" pod "4a2d6afc-c986-4ec7-85fb-1a8762bb4943" (UID: "4a2d6afc-c986-4ec7-85fb-1a8762bb4943"). InnerVolumeSpecName "kube-api-access-kghtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:21.691909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.691869 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kghtc\" (UniqueName: \"kubernetes.io/projected/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-kube-api-access-kghtc\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:21.691909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.691909 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:21.691909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:21.691920 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d6afc-c986-4ec7-85fb-1a8762bb4943-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:22.359669 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:22.359640 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" Oct 30 19:07:22.359847 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:22.359686 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8" event={"ID":"4a2d6afc-c986-4ec7-85fb-1a8762bb4943","Type":"ContainerDied","Data":"34010bcf8ba191075caa5d5b048c57bb1d2b1ae3519ce6615198ec336c005580"} Oct 30 19:07:22.359847 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:22.359769 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34010bcf8ba191075caa5d5b048c57bb1d2b1ae3519ce6615198ec336c005580" Oct 30 19:07:28.955720 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:28.955681 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d"] Oct 30 19:07:28.989757 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:28.989724 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q"] Oct 30 19:07:31.465402 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:31.465365 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8"] Oct 30 19:07:31.468818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:31.468778 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-861bb3-phtm8"] Oct 30 19:07:33.062609 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:33.062575 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2d6afc-c986-4ec7-85fb-1a8762bb4943" path="/var/lib/kubelet/pods/4a2d6afc-c986-4ec7-85fb-1a8762bb4943/volumes" Oct 30 19:07:33.957038 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:33.957001 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" podUID="41efe939-9a37-451e-b6d6-3a47877fbe13" containerName="payload-processing" containerID="cri-o://b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa" gracePeriod=25 Oct 30 19:07:33.990053 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:33.990019 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" podUID="b86db503-6cf0-4171-8020-a08639061418" containerName="payload-pre-processing" containerID="cri-o://5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511" gracePeriod=25 Oct 30 19:07:39.108931 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.108898 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7"] Oct 30 19:07:39.109532 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.109516 4862 cpu_manager.go:414] "containerMap: removing container" podUID="4a2d6afc-c986-4ec7-85fb-1a8762bb4943" containerName="revoke-keys" Oct 30 19:07:39.109583 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.109533 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2d6afc-c986-4ec7-85fb-1a8762bb4943" containerName="revoke-keys" Oct 30 19:07:39.109704 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.109693 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2d6afc-c986-4ec7-85fb-1a8762bb4943" containerName="revoke-keys" Oct 30 19:07:39.113461 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.113444 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.132704 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.132681 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7"] Oct 30 19:07:39.133415 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.133387 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.133564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.133543 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.133692 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.133675 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6j6d\" (UniqueName: \"kubernetes.io/projected/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-kube-api-access-x6j6d\") pod \"maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.174414 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.174381 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" podUID="41efe939-9a37-451e-b6d6-3a47877fbe13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:07:39.174414 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.174397 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" podUID="b86db503-6cf0-4171-8020-a08639061418" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:07:39.209489 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.209461 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:07:39.209755 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.209738 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:07:39.235031 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.235002 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.235149 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.235105 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.235218 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.235168 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6j6d\" (UniqueName: \"kubernetes.io/projected/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-kube-api-access-x6j6d\") pod \"maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.235686 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.235663 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.237695 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.237675 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.252921 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.252892 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6j6d\" (UniqueName: \"kubernetes.io/projected/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-kube-api-access-x6j6d\") pod \"maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.427217 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.427144 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:39.744466 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:39.744397 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7"] Oct 30 19:07:39.751037 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:07:39.751007 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba40ccb_26c3_461a_8a9b_888a5e3a9ee7.slice/crio-01c9214f08baa5740b6fe3dd67327a2fb2c236e46053f0cc1892c69be87c9055 WatchSource:0}: Error finding container 01c9214f08baa5740b6fe3dd67327a2fb2c236e46053f0cc1892c69be87c9055: Status 404 returned error can't find the container with id 01c9214f08baa5740b6fe3dd67327a2fb2c236e46053f0cc1892c69be87c9055 Oct 30 19:07:40.488888 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:40.488849 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" event={"ID":"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7","Type":"ContainerStarted","Data":"f9ee77d66dc2259a149153debd4f32d7bd425f8ddeb8adc26d69c436684c0358"} Oct 30 19:07:40.489324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:40.488918 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" event={"ID":"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7","Type":"ContainerStarted","Data":"01c9214f08baa5740b6fe3dd67327a2fb2c236e46053f0cc1892c69be87c9055"} Oct 30 19:07:40.528785 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:40.528736 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" podStartSLOduration=1.5287239719999999 podStartE2EDuration="1.528723972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:40.524855771 +0000 UTC m=+4159.573900683" watchObservedRunningTime="2026-10-30 19:07:40.528723972 +0000 UTC m=+4159.577768892" Oct 30 19:07:41.497672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:41.497644 4862 generic.go:356] "Generic (PLEG): container finished" podID="2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7" containerID="f9ee77d66dc2259a149153debd4f32d7bd425f8ddeb8adc26d69c436684c0358" exitCode=0 Oct 30 19:07:41.498061 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:41.497734 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" event={"ID":"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7","Type":"ContainerDied","Data":"f9ee77d66dc2259a149153debd4f32d7bd425f8ddeb8adc26d69c436684c0358"} Oct 30 19:07:42.662060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.662038 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:42.779397 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.779317 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-xks-ca\") pod \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " Oct 30 19:07:42.779548 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.779478 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-openshift-ca\") pod \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " Oct 30 19:07:42.779548 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.779534 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-kube-api-access-x6j6d\" (UniqueName: \"kubernetes.io/projected/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-kube-api-access-x6j6d\") pod \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\" (UID: \"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7\") " Oct 30 19:07:42.779848 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.779825 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-openshift-ca" pod "2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7" (UID: "2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:42.780087 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.780068 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:42.781564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.781544 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-xks-ca" pod "2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7" (UID: "2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:07:42.781843 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.781824 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-kube-api-access-x6j6d" pod "2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7" (UID: "2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7"). InnerVolumeSpecName "kube-api-access-x6j6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:42.881804 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.881779 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6j6d\" (UniqueName: \"kubernetes.io/projected/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-kube-api-access-x6j6d\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:42.881804 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:42.881804 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:43.513796 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:43.513720 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" Oct 30 19:07:43.513958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:43.513802 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7" event={"ID":"2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7","Type":"ContainerDied","Data":"01c9214f08baa5740b6fe3dd67327a2fb2c236e46053f0cc1892c69be87c9055"} Oct 30 19:07:43.513958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:43.513855 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01c9214f08baa5740b6fe3dd67327a2fb2c236e46053f0cc1892c69be87c9055" Oct 30 19:07:49.174852 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:49.174806 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" podUID="41efe939-9a37-451e-b6d6-3a47877fbe13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:07:49.175245 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:49.174860 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" podUID="b86db503-6cf0-4171-8020-a08639061418" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:07:49.610223 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:49.610193 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2"] Oct 30 19:07:49.903204 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:07:49.903115 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:07:50.040815 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:50.040776 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:07:53.832825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:53.832789 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7"] Oct 30 19:07:53.843013 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:53.842985 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6018b4-dgbq7"] Oct 30 19:07:54.284355 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.284333 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 19:07:54.314806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.314784 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 19:07:54.338516 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.338491 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41efe939-9a37-451e-b6d6-3a47877fbe13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41efe939-9a37-451e-b6d6-3a47877fbe13-plugins-config-volume\") pod \"41efe939-9a37-451e-b6d6-3a47877fbe13\" (UID: \"41efe939-9a37-451e-b6d6-3a47877fbe13\") " Oct 30 19:07:54.338793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.338771 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41efe939-9a37-451e-b6d6-3a47877fbe13-kube-api-access-4b2br\" (UniqueName: \"kubernetes.io/projected/41efe939-9a37-451e-b6d6-3a47877fbe13-kube-api-access-4b2br\") pod \"41efe939-9a37-451e-b6d6-3a47877fbe13\" (UID: \"41efe939-9a37-451e-b6d6-3a47877fbe13\") " Oct 30 19:07:54.338910 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.338890 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41efe939-9a37-451e-b6d6-3a47877fbe13-plugins-config-volume" pod "41efe939-9a37-451e-b6d6-3a47877fbe13" (UID: "41efe939-9a37-451e-b6d6-3a47877fbe13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:54.339287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.339249 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41efe939-9a37-451e-b6d6-3a47877fbe13-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:54.341082 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.341059 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41efe939-9a37-451e-b6d6-3a47877fbe13-kube-api-access-4b2br" pod "41efe939-9a37-451e-b6d6-3a47877fbe13" (UID: "41efe939-9a37-451e-b6d6-3a47877fbe13"). InnerVolumeSpecName "kube-api-access-4b2br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:54.440627 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.440535 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86db503-6cf0-4171-8020-a08639061418-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86db503-6cf0-4171-8020-a08639061418-plugins-config-volume\") pod \"b86db503-6cf0-4171-8020-a08639061418\" (UID: \"b86db503-6cf0-4171-8020-a08639061418\") " Oct 30 19:07:54.440795 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.440627 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86db503-6cf0-4171-8020-a08639061418-kube-api-access-b8pjg\" (UniqueName: \"kubernetes.io/projected/b86db503-6cf0-4171-8020-a08639061418-kube-api-access-b8pjg\") pod \"b86db503-6cf0-4171-8020-a08639061418\" (UID: \"b86db503-6cf0-4171-8020-a08639061418\") " Oct 30 19:07:54.440973 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.440948 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86db503-6cf0-4171-8020-a08639061418-plugins-config-volume" pod "b86db503-6cf0-4171-8020-a08639061418" (UID: "b86db503-6cf0-4171-8020-a08639061418"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:07:54.441182 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.441167 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b2br\" (UniqueName: \"kubernetes.io/projected/41efe939-9a37-451e-b6d6-3a47877fbe13-kube-api-access-4b2br\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:54.441460 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.441184 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86db503-6cf0-4171-8020-a08639061418-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:54.442921 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.442894 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86db503-6cf0-4171-8020-a08639061418-kube-api-access-b8pjg" pod "b86db503-6cf0-4171-8020-a08639061418" (UID: "b86db503-6cf0-4171-8020-a08639061418"). InnerVolumeSpecName "kube-api-access-b8pjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:07:54.542019 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.541993 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8pjg\" (UniqueName: \"kubernetes.io/projected/b86db503-6cf0-4171-8020-a08639061418-kube-api-access-b8pjg\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:07:54.590968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.590940 4862 generic.go:356] "Generic (PLEG): container finished" podID="b86db503-6cf0-4171-8020-a08639061418" containerID="5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511" exitCode=0 Oct 30 19:07:54.591108 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.590987 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" Oct 30 19:07:54.591108 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.591030 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" event={"ID":"b86db503-6cf0-4171-8020-a08639061418","Type":"ContainerDied","Data":"5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511"} Oct 30 19:07:54.591108 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.591091 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q" event={"ID":"b86db503-6cf0-4171-8020-a08639061418","Type":"ContainerDied","Data":"ec5a647cbd989a5ef82c041103d640372852cad6672b696675fd49713902588a"} Oct 30 19:07:54.591256 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.591125 4862 scope.go:122] "RemoveContainer" containerID="5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511" Oct 30 19:07:54.592498 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.592474 4862 generic.go:356] "Generic (PLEG): container finished" podID="41efe939-9a37-451e-b6d6-3a47877fbe13" containerID="b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa" exitCode=0 Oct 30 19:07:54.592612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.592503 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" event={"ID":"41efe939-9a37-451e-b6d6-3a47877fbe13","Type":"ContainerDied","Data":"b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa"} Oct 30 19:07:54.592612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.592528 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" Oct 30 19:07:54.592612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.592556 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d" event={"ID":"41efe939-9a37-451e-b6d6-3a47877fbe13","Type":"ContainerDied","Data":"879b08a54e77a7f9b394cb559411482d692017d4bd34b04d208fb19619311168"} Oct 30 19:07:54.610529 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.610497 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" podUID="579a0de2-def5-43cf-8fa4-7fb58cea0f4e" containerName="payload-processing" containerID="cri-o://adfd1dff8020ee79466e508138c34462fe0c0d864ebe10a705d5615ca7816184" gracePeriod=25 Oct 30 19:07:54.618469 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.618443 4862 scope.go:122] "RemoveContainer" containerID="5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511" Oct 30 19:07:54.618816 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:07:54.618785 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511\": container with ID starting with 5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511 not found: ID does not exist" containerID="5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511" Oct 30 19:07:54.618931 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.618825 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511"} err="failed to get container status \"5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511\": rpc error: code = NotFound desc = could not find container \"5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511\": container with ID starting with 5e1b4aee6ff6167f5e168abc4e7f3cded398af0a7a94e58126910c74291a1511 not found: ID does not exist" Oct 30 19:07:54.618931 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.618848 4862 scope.go:122] "RemoveContainer" containerID="b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa" Oct 30 19:07:54.631383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.630440 4862 scope.go:122] "RemoveContainer" containerID="b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa" Oct 30 19:07:54.631633 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:07:54.631611 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa\": container with ID starting with b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa not found: ID does not exist" containerID="b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa" Oct 30 19:07:54.631693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.631644 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa"} err="failed to get container status \"b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa\": rpc error: code = NotFound desc = could not find container \"b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa\": container with ID starting with b1cb6aa38fb3e6f23609f5f74e34c756f0dc13932404516ffde5243bbbd530aa not found: ID does not exist" Oct 30 19:07:54.639824 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.639801 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q"] Oct 30 19:07:54.643298 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.643278 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-861bb3-7bb884bf79-c4k7q"] Oct 30 19:07:54.686245 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.686218 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d"] Oct 30 19:07:54.689752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:54.689729 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-861bb3-589555cc9b-k659d"] Oct 30 19:07:55.062517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:55.062484 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7" path="/var/lib/kubelet/pods/2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7/volumes" Oct 30 19:07:55.062950 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:55.062933 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41efe939-9a37-451e-b6d6-3a47877fbe13" path="/var/lib/kubelet/pods/41efe939-9a37-451e-b6d6-3a47877fbe13/volumes" Oct 30 19:07:55.063381 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:55.063365 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86db503-6cf0-4171-8020-a08639061418" path="/var/lib/kubelet/pods/b86db503-6cf0-4171-8020-a08639061418/volumes" Oct 30 19:07:57.032333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:07:57.032300 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" podUID="579a0de2-def5-43cf-8fa4-7fb58cea0f4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:01.443870 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:01.443777 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q"] Oct 30 19:08:01.444636 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:01.444058 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" containerID="cri-o://a6c60e5fe5a3c611d3b889daee851dd785c0ed9bcefc566084452aac445c4fa1" gracePeriod=30 Oct 30 19:08:02.043523 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.043478 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t"] Oct 30 19:08:02.044065 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.044051 4862 cpu_manager.go:414] "containerMap: removing container" podUID="41efe939-9a37-451e-b6d6-3a47877fbe13" containerName="payload-processing" Oct 30 19:08:02.044135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.044067 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41efe939-9a37-451e-b6d6-3a47877fbe13" containerName="payload-processing" Oct 30 19:08:02.044135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.044083 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7" containerName="revoke-keys" Oct 30 19:08:02.044135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.044088 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7" containerName="revoke-keys" Oct 30 19:08:02.044135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.044124 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b86db503-6cf0-4171-8020-a08639061418" containerName="payload-pre-processing" Oct 30 19:08:02.044135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.044129 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86db503-6cf0-4171-8020-a08639061418" containerName="payload-pre-processing" Oct 30 19:08:02.044348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.044218 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="41efe939-9a37-451e-b6d6-3a47877fbe13" containerName="payload-processing" Oct 30 19:08:02.044348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.044229 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba40ccb-26c3-461a-8a9b-888a5e3a9ee7" containerName="revoke-keys" Oct 30 19:08:02.044348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.044240 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86db503-6cf0-4171-8020-a08639061418" containerName="payload-pre-processing" Oct 30 19:08:02.047868 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.047850 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.070312 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.069010 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t"] Oct 30 19:08:02.145844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.145813 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:08:02.146330 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.146311 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:08:02.236140 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.236108 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdsh8\" (UniqueName: \"kubernetes.io/projected/23cdd015-a0d3-40e8-b92b-5fb4650ca946-kube-api-access-qdsh8\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.236330 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.236167 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.236330 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.236304 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.338061 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.338027 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdsh8\" (UniqueName: \"kubernetes.io/projected/23cdd015-a0d3-40e8-b92b-5fb4650ca946-kube-api-access-qdsh8\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.338213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.338099 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.338213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.338159 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.338844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.338825 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.340521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.340503 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.357189 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.357157 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdsh8\" (UniqueName: \"kubernetes.io/projected/23cdd015-a0d3-40e8-b92b-5fb4650ca946-kube-api-access-qdsh8\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.361293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.361255 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:02.662968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:02.662941 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t"] Oct 30 19:08:02.665426 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:08:02.665400 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cdd015_a0d3_40e8_b92b_5fb4650ca946.slice/crio-bb1e704d189f57daf4ec893a695c64bece3c728644c4e224eaef638b5fe079af WatchSource:0}: Error finding container bb1e704d189f57daf4ec893a695c64bece3c728644c4e224eaef638b5fe079af: Status 404 returned error can't find the container with id bb1e704d189f57daf4ec893a695c64bece3c728644c4e224eaef638b5fe079af Oct 30 19:08:03.661029 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:03.660996 4862 generic.go:356] "Generic (PLEG): container finished" podID="23cdd015-a0d3-40e8-b92b-5fb4650ca946" containerID="4e02106610137241a7790ce973f0878536c1d5999916a13925363f590f81ced6" exitCode=0 Oct 30 19:08:03.661185 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:03.661075 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" event={"ID":"23cdd015-a0d3-40e8-b92b-5fb4650ca946","Type":"ContainerDied","Data":"4e02106610137241a7790ce973f0878536c1d5999916a13925363f590f81ced6"} Oct 30 19:08:03.661185 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:03.661133 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" event={"ID":"23cdd015-a0d3-40e8-b92b-5fb4650ca946","Type":"ContainerStarted","Data":"bb1e704d189f57daf4ec893a695c64bece3c728644c4e224eaef638b5fe079af"} Oct 30 19:08:04.907975 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:04.907945 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:05.025362 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.024904 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-xks-ca\") pod \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " Oct 30 19:08:05.025362 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.025048 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-openshift-ca\") pod \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " Oct 30 19:08:05.025362 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.025140 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cdd015-a0d3-40e8-b92b-5fb4650ca946-kube-api-access-qdsh8\" (UniqueName: \"kubernetes.io/projected/23cdd015-a0d3-40e8-b92b-5fb4650ca946-kube-api-access-qdsh8\") pod \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\" (UID: \"23cdd015-a0d3-40e8-b92b-5fb4650ca946\") " Oct 30 19:08:05.034721 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.032599 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-openshift-ca" pod "23cdd015-a0d3-40e8-b92b-5fb4650ca946" (UID: "23cdd015-a0d3-40e8-b92b-5fb4650ca946"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:05.037314 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.035967 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cdd015-a0d3-40e8-b92b-5fb4650ca946-kube-api-access-qdsh8" pod "23cdd015-a0d3-40e8-b92b-5fb4650ca946" (UID: "23cdd015-a0d3-40e8-b92b-5fb4650ca946"). InnerVolumeSpecName "kube-api-access-qdsh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:05.037314 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.037307 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-xks-ca" pod "23cdd015-a0d3-40e8-b92b-5fb4650ca946" (UID: "23cdd015-a0d3-40e8-b92b-5fb4650ca946"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:05.127310 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.127249 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:05.127310 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.127309 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdsh8\" (UniqueName: \"kubernetes.io/projected/23cdd015-a0d3-40e8-b92b-5fb4650ca946-kube-api-access-qdsh8\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:05.127519 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.127325 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdd015-a0d3-40e8-b92b-5fb4650ca946-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:05.676564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.676526 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" event={"ID":"23cdd015-a0d3-40e8-b92b-5fb4650ca946","Type":"ContainerDied","Data":"bb1e704d189f57daf4ec893a695c64bece3c728644c4e224eaef638b5fe079af"} Oct 30 19:08:05.676564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.676564 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t" Oct 30 19:08:05.676769 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.676585 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb1e704d189f57daf4ec893a695c64bece3c728644c4e224eaef638b5fe079af" Oct 30 19:08:05.756201 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.756169 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9"] Oct 30 19:08:05.759299 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:05.759275 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2107d4-jk2l9"] Oct 30 19:08:06.688202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.688171 4862 generic.go:356] "Generic (PLEG): container finished" podID="6a741c85-d6fc-4bac-82c1-58884544c340" containerID="a6c60e5fe5a3c611d3b889daee851dd785c0ed9bcefc566084452aac445c4fa1" exitCode=0 Oct 30 19:08:06.688693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.688242 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" event={"ID":"6a741c85-d6fc-4bac-82c1-58884544c340","Type":"ContainerDied","Data":"a6c60e5fe5a3c611d3b889daee851dd785c0ed9bcefc566084452aac445c4fa1"} Oct 30 19:08:06.688693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.688347 4862 scope.go:122] "RemoveContainer" containerID="366f11bbdc5f36eebbeabbe5400169b0b9c58a78da49e93a27eac5726b9c54ff" Oct 30 19:08:06.723175 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.723157 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 19:08:06.847653 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.847627 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-envoy\") pod \"6a741c85-d6fc-4bac-82c1-58884544c340\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " Oct 30 19:08:06.847816 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.847680 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a741c85-d6fc-4bac-82c1-58884544c340-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a741c85-d6fc-4bac-82c1-58884544c340-istiod-ca-cert\") pod \"6a741c85-d6fc-4bac-82c1-58884544c340\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " Oct 30 19:08:06.847816 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.847780 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-data\") pod \"6a741c85-d6fc-4bac-82c1-58884544c340\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " Oct 30 19:08:06.847933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.847836 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-istio-token\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-istio-token\") pod \"6a741c85-d6fc-4bac-82c1-58884544c340\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " Oct 30 19:08:06.847933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.847896 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-socket\") pod \"6a741c85-d6fc-4bac-82c1-58884544c340\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " Oct 30 19:08:06.848050 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.847963 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-kube-api-access-tcqh8\") pod \"6a741c85-d6fc-4bac-82c1-58884544c340\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " Oct 30 19:08:06.848050 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.848010 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a741c85-d6fc-4bac-82c1-58884544c340-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a741c85-d6fc-4bac-82c1-58884544c340-istio-podinfo\") pod \"6a741c85-d6fc-4bac-82c1-58884544c340\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " Oct 30 19:08:06.848162 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.848083 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-credential-socket\") pod \"6a741c85-d6fc-4bac-82c1-58884544c340\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " Oct 30 19:08:06.848162 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.848118 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-data" pod "6a741c85-d6fc-4bac-82c1-58884544c340" (UID: "6a741c85-d6fc-4bac-82c1-58884544c340"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:06.848162 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.848123 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-certs\") pod \"6a741c85-d6fc-4bac-82c1-58884544c340\" (UID: \"6a741c85-d6fc-4bac-82c1-58884544c340\") " Oct 30 19:08:06.848354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.848158 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-socket" pod "6a741c85-d6fc-4bac-82c1-58884544c340" (UID: "6a741c85-d6fc-4bac-82c1-58884544c340"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:06.848354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.848176 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a741c85-d6fc-4bac-82c1-58884544c340-istiod-ca-cert" pod "6a741c85-d6fc-4bac-82c1-58884544c340" (UID: "6a741c85-d6fc-4bac-82c1-58884544c340"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:06.848487 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.848357 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-certs" pod "6a741c85-d6fc-4bac-82c1-58884544c340" (UID: "6a741c85-d6fc-4bac-82c1-58884544c340"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:06.848597 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.848543 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-credential-socket" pod "6a741c85-d6fc-4bac-82c1-58884544c340" (UID: "6a741c85-d6fc-4bac-82c1-58884544c340"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:06.849022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.848996 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:06.849135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.849028 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a741c85-d6fc-4bac-82c1-58884544c340-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:06.849135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.849044 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:06.849135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.849057 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:06.849135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.849071 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:06.850123 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.850099 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-envoy" pod "6a741c85-d6fc-4bac-82c1-58884544c340" (UID: "6a741c85-d6fc-4bac-82c1-58884544c340"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:08:06.850293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.850249 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-istio-token" pod "6a741c85-d6fc-4bac-82c1-58884544c340" (UID: "6a741c85-d6fc-4bac-82c1-58884544c340"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:06.850353 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.850321 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a741c85-d6fc-4bac-82c1-58884544c340-istio-podinfo" pod "6a741c85-d6fc-4bac-82c1-58884544c340" (UID: "6a741c85-d6fc-4bac-82c1-58884544c340"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:08:06.850580 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.850554 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-kube-api-access-tcqh8" pod "6a741c85-d6fc-4bac-82c1-58884544c340" (UID: "6a741c85-d6fc-4bac-82c1-58884544c340"). InnerVolumeSpecName "kube-api-access-tcqh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:06.950356 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.950338 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a741c85-d6fc-4bac-82c1-58884544c340-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:06.950356 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.950357 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:06.950494 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.950368 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqh8\" (UniqueName: \"kubernetes.io/projected/6a741c85-d6fc-4bac-82c1-58884544c340-kube-api-access-tcqh8\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:06.950494 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:06.950377 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a741c85-d6fc-4bac-82c1-58884544c340-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:07.032665 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:07.032599 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" podUID="579a0de2-def5-43cf-8fa4-7fb58cea0f4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:07.063144 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:07.063116 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5304b2-799f-431e-bc86-563c681ebc69" path="/var/lib/kubelet/pods/3a5304b2-799f-431e-bc86-563c681ebc69/volumes" Oct 30 19:08:07.696729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:07.696702 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" Oct 30 19:08:07.697176 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:07.696741 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q" event={"ID":"6a741c85-d6fc-4bac-82c1-58884544c340","Type":"ContainerDied","Data":"dee3e216c1eedec19cae027ef94f54a6d00de4e50dad1bbee823da789001631b"} Oct 30 19:08:07.697176 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:07.696791 4862 scope.go:122] "RemoveContainer" containerID="a6c60e5fe5a3c611d3b889daee851dd785c0ed9bcefc566084452aac445c4fa1" Oct 30 19:08:07.783950 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:07.783916 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q"] Oct 30 19:08:07.791724 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:07.791698 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6018b4-openshift-default-7fd78b55cc-pl28q"] Oct 30 19:08:09.063179 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:09.063146 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" path="/var/lib/kubelet/pods/6a741c85-d6fc-4bac-82c1-58884544c340/volumes" Oct 30 19:08:14.704496 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.704467 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd"] Oct 30 19:08:14.705060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.705046 4862 cpu_manager.go:414] "containerMap: removing container" podUID="23cdd015-a0d3-40e8-b92b-5fb4650ca946" containerName="revoke-keys" Oct 30 19:08:14.705060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.705061 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cdd015-a0d3-40e8-b92b-5fb4650ca946" containerName="revoke-keys" Oct 30 19:08:14.705184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.705076 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" Oct 30 19:08:14.705184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.705081 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" Oct 30 19:08:14.705319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.705216 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" Oct 30 19:08:14.705319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.705233 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cdd015-a0d3-40e8-b92b-5fb4650ca946" containerName="revoke-keys" Oct 30 19:08:14.708723 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.708707 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:14.732848 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.732801 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd"] Oct 30 19:08:14.760286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.759521 4862 generic.go:356] "Generic (PLEG): container finished" podID="579a0de2-def5-43cf-8fa4-7fb58cea0f4e" containerID="adfd1dff8020ee79466e508138c34462fe0c0d864ebe10a705d5615ca7816184" exitCode=0 Oct 30 19:08:14.760286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.759573 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" event={"ID":"579a0de2-def5-43cf-8fa4-7fb58cea0f4e","Type":"ContainerDied","Data":"adfd1dff8020ee79466e508138c34462fe0c0d864ebe10a705d5615ca7816184"} Oct 30 19:08:14.769405 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.769374 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:08:14.769615 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.769512 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:08:14.889188 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.887865 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:14.889188 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.888006 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:14.889188 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.888087 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zmg\" (UniqueName: \"kubernetes.io/projected/f1521770-63e7-40e3-834c-d5df45f8afa3-kube-api-access-p9zmg\") pod \"maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:14.938000 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.937977 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 19:08:14.990666 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.990636 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:14.990890 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.990777 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:14.990890 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.990857 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zmg\" (UniqueName: \"kubernetes.io/projected/f1521770-63e7-40e3-834c-d5df45f8afa3-kube-api-access-p9zmg\") pod \"maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:14.991597 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.991564 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:14.994014 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:14.993948 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:15.009532 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.009504 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zmg\" (UniqueName: \"kubernetes.io/projected/f1521770-63e7-40e3-834c-d5df45f8afa3-kube-api-access-p9zmg\") pod \"maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:15.026022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.026003 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:15.091823 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.091792 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-plugins-config-volume\") pod \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\" (UID: \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\") " Oct 30 19:08:15.091976 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.091957 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-kube-api-access-cnwd7\" (UniqueName: \"kubernetes.io/projected/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-kube-api-access-cnwd7\") pod \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\" (UID: \"579a0de2-def5-43cf-8fa4-7fb58cea0f4e\") " Oct 30 19:08:15.094022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.093462 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-plugins-config-volume" pod "579a0de2-def5-43cf-8fa4-7fb58cea0f4e" (UID: "579a0de2-def5-43cf-8fa4-7fb58cea0f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:15.103488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.103423 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-kube-api-access-cnwd7" pod "579a0de2-def5-43cf-8fa4-7fb58cea0f4e" (UID: "579a0de2-def5-43cf-8fa4-7fb58cea0f4e"). InnerVolumeSpecName "kube-api-access-cnwd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:15.194325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.194225 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:15.194325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.194257 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwd7\" (UniqueName: \"kubernetes.io/projected/579a0de2-def5-43cf-8fa4-7fb58cea0f4e-kube-api-access-cnwd7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:15.341004 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.340968 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd"] Oct 30 19:08:15.344636 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:08:15.344607 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1521770_63e7_40e3_834c_d5df45f8afa3.slice/crio-e6f7ae17ac56d9cd1791511e6b5d2c86463141797e03835fdd47325d6e90eeba WatchSource:0}: Error finding container e6f7ae17ac56d9cd1791511e6b5d2c86463141797e03835fdd47325d6e90eeba: Status 404 returned error can't find the container with id e6f7ae17ac56d9cd1791511e6b5d2c86463141797e03835fdd47325d6e90eeba Oct 30 19:08:15.530350 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.530282 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2"] Oct 30 19:08:15.530913 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.530895 4862 cpu_manager.go:414] "containerMap: removing container" podUID="579a0de2-def5-43cf-8fa4-7fb58cea0f4e" containerName="payload-processing" Oct 30 19:08:15.530962 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.530918 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579a0de2-def5-43cf-8fa4-7fb58cea0f4e" containerName="payload-processing" Oct 30 19:08:15.531014 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.530961 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" Oct 30 19:08:15.531014 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.530967 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" Oct 30 19:08:15.531147 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.531129 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a741c85-d6fc-4bac-82c1-58884544c340" containerName="istio-proxy" Oct 30 19:08:15.531226 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.531157 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="579a0de2-def5-43cf-8fa4-7fb58cea0f4e" containerName="payload-processing" Oct 30 19:08:15.536216 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.536189 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.555494 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.555469 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1a2436\"/\"rate-test-model-1a2436-kserve-self-signed-certs\"" Oct 30 19:08:15.555640 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.555614 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1a2436\"/\"default-dockercfg-kz6gz\"" Oct 30 19:08:15.558026 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.556236 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1a2436\"/\"openshift-service-ca.crt\"" Oct 30 19:08:15.558026 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.556351 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1a2436\"/\"kube-root-ca.crt\"" Oct 30 19:08:15.562966 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.562942 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2"] Oct 30 19:08:15.699786 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.699750 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-dshm\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.699938 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.699907 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-home\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.699982 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.699953 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7508ead-c54b-4b59-a49b-46df7370cac2-tls-certs\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.700052 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.700026 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvnql\" (UniqueName: \"kubernetes.io/projected/b7508ead-c54b-4b59-a49b-46df7370cac2-kube-api-access-vvnql\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.700224 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.700209 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-model-cache\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.700278 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.700246 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-tmp-dir\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.784333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.778410 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" event={"ID":"579a0de2-def5-43cf-8fa4-7fb58cea0f4e","Type":"ContainerDied","Data":"1fe0c18495ed7bd4eeba1770091afa3293d298a937c043eff1998b6c11cea19a"} Oct 30 19:08:15.784333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.778488 4862 scope.go:122] "RemoveContainer" containerID="adfd1dff8020ee79466e508138c34462fe0c0d864ebe10a705d5615ca7816184" Oct 30 19:08:15.784333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.778664 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2" Oct 30 19:08:15.787310 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.785601 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" event={"ID":"f1521770-63e7-40e3-834c-d5df45f8afa3","Type":"ContainerStarted","Data":"d42ce3a2c0c50f32144cc930e7740fb9e76374b8591b7895dad704e10fae253c"} Oct 30 19:08:15.787310 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.785660 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" event={"ID":"f1521770-63e7-40e3-834c-d5df45f8afa3","Type":"ContainerStarted","Data":"e6f7ae17ac56d9cd1791511e6b5d2c86463141797e03835fdd47325d6e90eeba"} Oct 30 19:08:15.813507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.806467 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-model-cache\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.813507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.806522 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-tmp-dir\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.813507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.806607 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-dshm\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.813507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.806699 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-home\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.813507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.806742 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7508ead-c54b-4b59-a49b-46df7370cac2-tls-certs\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.813507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.806767 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvnql\" (UniqueName: \"kubernetes.io/projected/b7508ead-c54b-4b59-a49b-46df7370cac2-kube-api-access-vvnql\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.813507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.806889 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-model-cache\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.813507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.807453 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-tmp-dir\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.813507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.809657 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-home\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.817022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.816994 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-dshm\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.817893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.817851 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7508ead-c54b-4b59-a49b-46df7370cac2-tls-certs\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.827696 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.826801 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvnql\" (UniqueName: \"kubernetes.io/projected/b7508ead-c54b-4b59-a49b-46df7370cac2-kube-api-access-vvnql\") pod \"rate-test-model-1a2436-kserve-6f45766669-49vp2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.861094 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.861062 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:15.873896 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.873838 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" podStartSLOduration=1.873821653 podStartE2EDuration="1.873821653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:15.838849317 +0000 UTC m=+4194.887894231" watchObservedRunningTime="2026-10-30 19:08:15.873821653 +0000 UTC m=+4194.922866568" Oct 30 19:08:15.875411 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.875346 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2"] Oct 30 19:08:15.880551 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:15.880409 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6018b4-f55df5fb8-4gsb2"] Oct 30 19:08:16.191775 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:16.191747 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2"] Oct 30 19:08:16.214620 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:08:16.214596 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7508ead_c54b_4b59_a49b_46df7370cac2.slice/crio-dbc46a85534efddf01d4b5f4eca938f8ac31c873a3061cbe4fbd1d4a07353323 WatchSource:0}: Error finding container dbc46a85534efddf01d4b5f4eca938f8ac31c873a3061cbe4fbd1d4a07353323: Status 404 returned error can't find the container with id dbc46a85534efddf01d4b5f4eca938f8ac31c873a3061cbe4fbd1d4a07353323 Oct 30 19:08:16.795348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:16.795310 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" event={"ID":"b7508ead-c54b-4b59-a49b-46df7370cac2","Type":"ContainerStarted","Data":"0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820"} Oct 30 19:08:16.795741 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:16.795366 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" event={"ID":"b7508ead-c54b-4b59-a49b-46df7370cac2","Type":"ContainerStarted","Data":"dbc46a85534efddf01d4b5f4eca938f8ac31c873a3061cbe4fbd1d4a07353323"} Oct 30 19:08:16.797994 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:16.797965 4862 generic.go:356] "Generic (PLEG): container finished" podID="f1521770-63e7-40e3-834c-d5df45f8afa3" containerID="d42ce3a2c0c50f32144cc930e7740fb9e76374b8591b7895dad704e10fae253c" exitCode=0 Oct 30 19:08:16.798099 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:16.798046 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" event={"ID":"f1521770-63e7-40e3-834c-d5df45f8afa3","Type":"ContainerDied","Data":"d42ce3a2c0c50f32144cc930e7740fb9e76374b8591b7895dad704e10fae253c"} Oct 30 19:08:16.861424 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:16.861397 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:16.867031 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:16.866983 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" podStartSLOduration=1.8669692119999999 podStartE2EDuration="1.866969212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:16.832977038 +0000 UTC m=+4195.882021952" watchObservedRunningTime="2026-10-30 19:08:16.866969212 +0000 UTC m=+4195.916014128" Oct 30 19:08:17.063236 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:17.063205 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="579a0de2-def5-43cf-8fa4-7fb58cea0f4e" path="/var/lib/kubelet/pods/579a0de2-def5-43cf-8fa4-7fb58cea0f4e/volumes" Oct 30 19:08:17.951445 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:17.951420 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:18.135471 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.135405 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-xks-ca\") pod \"f1521770-63e7-40e3-834c-d5df45f8afa3\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " Oct 30 19:08:18.135471 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.135465 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1521770-63e7-40e3-834c-d5df45f8afa3-kube-api-access-p9zmg\" (UniqueName: \"kubernetes.io/projected/f1521770-63e7-40e3-834c-d5df45f8afa3-kube-api-access-p9zmg\") pod \"f1521770-63e7-40e3-834c-d5df45f8afa3\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " Oct 30 19:08:18.135660 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.135573 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-openshift-ca\") pod \"f1521770-63e7-40e3-834c-d5df45f8afa3\" (UID: \"f1521770-63e7-40e3-834c-d5df45f8afa3\") " Oct 30 19:08:18.135899 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.135868 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-openshift-ca" pod "f1521770-63e7-40e3-834c-d5df45f8afa3" (UID: "f1521770-63e7-40e3-834c-d5df45f8afa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:18.136182 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.136106 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:18.137777 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.137749 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-xks-ca" pod "f1521770-63e7-40e3-834c-d5df45f8afa3" (UID: "f1521770-63e7-40e3-834c-d5df45f8afa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:18.137909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.137886 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1521770-63e7-40e3-834c-d5df45f8afa3-kube-api-access-p9zmg" pod "f1521770-63e7-40e3-834c-d5df45f8afa3" (UID: "f1521770-63e7-40e3-834c-d5df45f8afa3"). InnerVolumeSpecName "kube-api-access-p9zmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:18.237615 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.237586 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1521770-63e7-40e3-834c-d5df45f8afa3-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:18.237615 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.237617 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9zmg\" (UniqueName: \"kubernetes.io/projected/f1521770-63e7-40e3-834c-d5df45f8afa3-kube-api-access-p9zmg\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:18.813253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.813216 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" Oct 30 19:08:18.813253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.813231 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd" event={"ID":"f1521770-63e7-40e3-834c-d5df45f8afa3","Type":"ContainerDied","Data":"e6f7ae17ac56d9cd1791511e6b5d2c86463141797e03835fdd47325d6e90eeba"} Oct 30 19:08:18.813492 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:18.813316 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6f7ae17ac56d9cd1791511e6b5d2c86463141797e03835fdd47325d6e90eeba" Oct 30 19:08:19.872653 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:08:19.872618 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:08:22.017632 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:22.017601 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 19:08:22.024424 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:22.024402 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 19:08:25.172425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:25.172394 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd"] Oct 30 19:08:25.861755 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:25.861721 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:25.882987 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:25.882955 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:26.873500 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:26.873469 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:08:30.173020 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:30.172976 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" podUID="2c3a2fa1-7a97-488b-8420-e9074e752891" containerName="payload-processing" containerID="cri-o://385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8" gracePeriod=25 Oct 30 19:08:30.939085 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:30.939047 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:08:32.224875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.224832 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd"] Oct 30 19:08:32.234451 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.234427 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-38e0db-j5kcd"] Oct 30 19:08:32.387400 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.387368 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n"] Oct 30 19:08:32.388631 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.388590 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f1521770-63e7-40e3-834c-d5df45f8afa3" containerName="revoke-keys" Oct 30 19:08:32.388820 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.388805 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1521770-63e7-40e3-834c-d5df45f8afa3" containerName="revoke-keys" Oct 30 19:08:32.389211 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.389196 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1521770-63e7-40e3-834c-d5df45f8afa3" containerName="revoke-keys" Oct 30 19:08:32.394942 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.394920 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.415232 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.412773 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:08:32.416128 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.416104 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:08:32.429328 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.429297 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n"] Oct 30 19:08:32.546650 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.546621 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.546834 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.546688 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.546925 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.546901 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrllx\" (UniqueName: \"kubernetes.io/projected/ac9b84d3-de21-4ff9-b515-02a7cdb79557-kube-api-access-vrllx\") pod \"maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.648683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.648654 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrllx\" (UniqueName: \"kubernetes.io/projected/ac9b84d3-de21-4ff9-b515-02a7cdb79557-kube-api-access-vrllx\") pod \"maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.648840 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.648740 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.648840 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.648781 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.649442 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.649423 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.651207 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.651187 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.673352 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.673328 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrllx\" (UniqueName: \"kubernetes.io/projected/ac9b84d3-de21-4ff9-b515-02a7cdb79557-kube-api-access-vrllx\") pod \"maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:32.749694 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:32.749672 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:33.011107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:33.011071 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" podUID="2c3a2fa1-7a97-488b-8420-e9074e752891" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:33.049543 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:33.049477 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n"] Oct 30 19:08:33.052608 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:08:33.052571 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9b84d3_de21_4ff9_b515_02a7cdb79557.slice/crio-58f0edd6e85716ac0497a8ffc460387927f55bd242ee8642dce3776b949b547f WatchSource:0}: Error finding container 58f0edd6e85716ac0497a8ffc460387927f55bd242ee8642dce3776b949b547f: Status 404 returned error can't find the container with id 58f0edd6e85716ac0497a8ffc460387927f55bd242ee8642dce3776b949b547f Oct 30 19:08:33.063964 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:33.063935 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1521770-63e7-40e3-834c-d5df45f8afa3" path="/var/lib/kubelet/pods/f1521770-63e7-40e3-834c-d5df45f8afa3/volumes" Oct 30 19:08:33.940110 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:33.940078 4862 generic.go:356] "Generic (PLEG): container finished" podID="ac9b84d3-de21-4ff9-b515-02a7cdb79557" containerID="6bd005c3a41d0f6cac89fbca033459e910040d56b3740fad49c8e241d6970261" exitCode=0 Oct 30 19:08:33.940478 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:33.940162 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" event={"ID":"ac9b84d3-de21-4ff9-b515-02a7cdb79557","Type":"ContainerDied","Data":"6bd005c3a41d0f6cac89fbca033459e910040d56b3740fad49c8e241d6970261"} Oct 30 19:08:33.940478 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:33.940217 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" event={"ID":"ac9b84d3-de21-4ff9-b515-02a7cdb79557","Type":"ContainerStarted","Data":"58f0edd6e85716ac0497a8ffc460387927f55bd242ee8642dce3776b949b547f"} Oct 30 19:08:35.113253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.113229 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:35.287871 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.287800 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-openshift-ca\") pod \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " Oct 30 19:08:35.287871 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.287863 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9b84d3-de21-4ff9-b515-02a7cdb79557-kube-api-access-vrllx\" (UniqueName: \"kubernetes.io/projected/ac9b84d3-de21-4ff9-b515-02a7cdb79557-kube-api-access-vrllx\") pod \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " Oct 30 19:08:35.288074 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.288028 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-xks-ca\") pod \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\" (UID: \"ac9b84d3-de21-4ff9-b515-02a7cdb79557\") " Oct 30 19:08:35.288153 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.288130 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-openshift-ca" pod "ac9b84d3-de21-4ff9-b515-02a7cdb79557" (UID: "ac9b84d3-de21-4ff9-b515-02a7cdb79557"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:35.288695 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.288607 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:35.290069 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.290047 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9b84d3-de21-4ff9-b515-02a7cdb79557-kube-api-access-vrllx" pod "ac9b84d3-de21-4ff9-b515-02a7cdb79557" (UID: "ac9b84d3-de21-4ff9-b515-02a7cdb79557"). InnerVolumeSpecName "kube-api-access-vrllx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:35.290181 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.290098 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-xks-ca" pod "ac9b84d3-de21-4ff9-b515-02a7cdb79557" (UID: "ac9b84d3-de21-4ff9-b515-02a7cdb79557"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:35.390082 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.390056 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrllx\" (UniqueName: \"kubernetes.io/projected/ac9b84d3-de21-4ff9-b515-02a7cdb79557-kube-api-access-vrllx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:35.390082 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.390080 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac9b84d3-de21-4ff9-b515-02a7cdb79557-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:35.956819 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.956776 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" event={"ID":"ac9b84d3-de21-4ff9-b515-02a7cdb79557","Type":"ContainerDied","Data":"58f0edd6e85716ac0497a8ffc460387927f55bd242ee8642dce3776b949b547f"} Oct 30 19:08:35.956819 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.956816 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n" Oct 30 19:08:35.957117 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:35.956835 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58f0edd6e85716ac0497a8ffc460387927f55bd242ee8642dce3776b949b547f" Oct 30 19:08:36.766951 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:36.766918 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t"] Oct 30 19:08:36.776738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:36.776713 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-e95c51-rqh2t"] Oct 30 19:08:37.062455 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:37.062422 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23cdd015-a0d3-40e8-b92b-5fb4650ca946" path="/var/lib/kubelet/pods/23cdd015-a0d3-40e8-b92b-5fb4650ca946/volumes" Oct 30 19:08:40.589557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.589519 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5"] Oct 30 19:08:40.590429 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.590407 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ac9b84d3-de21-4ff9-b515-02a7cdb79557" containerName="revoke-keys" Oct 30 19:08:40.590473 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.590432 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9b84d3-de21-4ff9-b515-02a7cdb79557" containerName="revoke-keys" Oct 30 19:08:40.590656 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.590644 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9b84d3-de21-4ff9-b515-02a7cdb79557" containerName="revoke-keys" Oct 30 19:08:40.596366 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.596343 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.609568 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.609540 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:08:40.612686 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.612661 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5"] Oct 30 19:08:40.615928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.615890 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:08:40.713189 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.713153 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.713412 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.713315 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.713485 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.713448 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rns6\" (UniqueName: \"kubernetes.io/projected/6a753f62-84e2-4ba1-a32b-f46093e2f672-kube-api-access-4rns6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.815062 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.815029 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.815305 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.815141 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rns6\" (UniqueName: \"kubernetes.io/projected/6a753f62-84e2-4ba1-a32b-f46093e2f672-kube-api-access-4rns6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.815305 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.815252 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.815778 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.815757 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.817820 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.817796 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.834207 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.834180 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rns6\" (UniqueName: \"kubernetes.io/projected/6a753f62-84e2-4ba1-a32b-f46093e2f672-kube-api-access-4rns6\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:40.916465 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:40.916377 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:41.228257 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:41.228234 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5"] Oct 30 19:08:41.230566 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:08:41.230534 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a753f62_84e2_4ba1_a32b_f46093e2f672.slice/crio-615221409e9a0080e5885accbe6606bcab0167f2fafb017179771daa943ec2b5 WatchSource:0}: Error finding container 615221409e9a0080e5885accbe6606bcab0167f2fafb017179771daa943ec2b5: Status 404 returned error can't find the container with id 615221409e9a0080e5885accbe6606bcab0167f2fafb017179771daa943ec2b5 Oct 30 19:08:42.002656 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:42.002622 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" event={"ID":"6a753f62-84e2-4ba1-a32b-f46093e2f672","Type":"ContainerStarted","Data":"26a76ca8a696fa25a13677e649bbc7ccab350a9732deb7e272c00f7f06a849af"} Oct 30 19:08:42.003061 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:42.002679 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" event={"ID":"6a753f62-84e2-4ba1-a32b-f46093e2f672","Type":"ContainerStarted","Data":"615221409e9a0080e5885accbe6606bcab0167f2fafb017179771daa943ec2b5"} Oct 30 19:08:42.042162 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:42.042117 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" podStartSLOduration=2.042103113 podStartE2EDuration="2.042103113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:42.037907622 +0000 UTC m=+4221.086952535" watchObservedRunningTime="2026-10-30 19:08:42.042103113 +0000 UTC m=+4221.091148026" Oct 30 19:08:42.550360 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:42.550326 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb"] Oct 30 19:08:42.584630 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:42.584598 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6"] Oct 30 19:08:43.010752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:43.010654 4862 generic.go:356] "Generic (PLEG): container finished" podID="6a753f62-84e2-4ba1-a32b-f46093e2f672" containerID="26a76ca8a696fa25a13677e649bbc7ccab350a9732deb7e272c00f7f06a849af" exitCode=0 Oct 30 19:08:43.011188 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:43.010839 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" event={"ID":"6a753f62-84e2-4ba1-a32b-f46093e2f672","Type":"ContainerDied","Data":"26a76ca8a696fa25a13677e649bbc7ccab350a9732deb7e272c00f7f06a849af"} Oct 30 19:08:43.011243 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:43.011178 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" podUID="2c3a2fa1-7a97-488b-8420-e9074e752891" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:08:44.183558 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.183533 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:44.258060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.258027 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-xks-ca\") pod \"6a753f62-84e2-4ba1-a32b-f46093e2f672\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " Oct 30 19:08:44.258239 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.258115 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-openshift-ca\") pod \"6a753f62-84e2-4ba1-a32b-f46093e2f672\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " Oct 30 19:08:44.258366 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.258335 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a753f62-84e2-4ba1-a32b-f46093e2f672-kube-api-access-4rns6\" (UniqueName: \"kubernetes.io/projected/6a753f62-84e2-4ba1-a32b-f46093e2f672-kube-api-access-4rns6\") pod \"6a753f62-84e2-4ba1-a32b-f46093e2f672\" (UID: \"6a753f62-84e2-4ba1-a32b-f46093e2f672\") " Oct 30 19:08:44.258543 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.258516 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-openshift-ca" pod "6a753f62-84e2-4ba1-a32b-f46093e2f672" (UID: "6a753f62-84e2-4ba1-a32b-f46093e2f672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:44.258954 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.258871 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:44.260440 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.260416 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-xks-ca" pod "6a753f62-84e2-4ba1-a32b-f46093e2f672" (UID: "6a753f62-84e2-4ba1-a32b-f46093e2f672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:44.260980 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.260949 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a753f62-84e2-4ba1-a32b-f46093e2f672-kube-api-access-4rns6" pod "6a753f62-84e2-4ba1-a32b-f46093e2f672" (UID: "6a753f62-84e2-4ba1-a32b-f46093e2f672"). InnerVolumeSpecName "kube-api-access-4rns6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:44.360453 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.360422 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rns6\" (UniqueName: \"kubernetes.io/projected/6a753f62-84e2-4ba1-a32b-f46093e2f672-kube-api-access-4rns6\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:44.360453 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:44.360449 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a753f62-84e2-4ba1-a32b-f46093e2f672-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:45.026152 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:45.026117 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" event={"ID":"6a753f62-84e2-4ba1-a32b-f46093e2f672","Type":"ContainerDied","Data":"615221409e9a0080e5885accbe6606bcab0167f2fafb017179771daa943ec2b5"} Oct 30 19:08:45.026324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:45.026176 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="615221409e9a0080e5885accbe6606bcab0167f2fafb017179771daa943ec2b5" Oct 30 19:08:45.026324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:45.026135 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5" Oct 30 19:08:47.551363 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:47.551323 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" podUID="fa302575-37dc-4227-a316-56258980eb26" containerName="payload-processing" containerID="cri-o://71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a" gracePeriod=25 Oct 30 19:08:47.585546 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:47.585519 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" podUID="0b124a84-203f-4628-96bd-4691396ffafc" containerName="payload-pre-processing" containerID="cri-o://5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd" gracePeriod=25 Oct 30 19:08:47.862651 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:47.862620 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:08:47.902609 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:47.902586 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:08:47.906606 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:47.906575 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa302575-37dc-4227-a316-56258980eb26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa302575-37dc-4227-a316-56258980eb26-plugins-config-volume\") pod \"fa302575-37dc-4227-a316-56258980eb26\" (UID: \"fa302575-37dc-4227-a316-56258980eb26\") " Oct 30 19:08:47.906711 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:47.906667 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa302575-37dc-4227-a316-56258980eb26-kube-api-access-54bvv\" (UniqueName: \"kubernetes.io/projected/fa302575-37dc-4227-a316-56258980eb26-kube-api-access-54bvv\") pod \"fa302575-37dc-4227-a316-56258980eb26\" (UID: \"fa302575-37dc-4227-a316-56258980eb26\") " Oct 30 19:08:47.906970 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:47.906948 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa302575-37dc-4227-a316-56258980eb26-plugins-config-volume" pod "fa302575-37dc-4227-a316-56258980eb26" (UID: "fa302575-37dc-4227-a316-56258980eb26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:47.907364 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:47.907330 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa302575-37dc-4227-a316-56258980eb26-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:47.908931 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:47.908903 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa302575-37dc-4227-a316-56258980eb26-kube-api-access-54bvv" pod "fa302575-37dc-4227-a316-56258980eb26" (UID: "fa302575-37dc-4227-a316-56258980eb26"). InnerVolumeSpecName "kube-api-access-54bvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:48.008485 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.008454 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b124a84-203f-4628-96bd-4691396ffafc-kube-api-access-ztj74\" (UniqueName: \"kubernetes.io/projected/0b124a84-203f-4628-96bd-4691396ffafc-kube-api-access-ztj74\") pod \"0b124a84-203f-4628-96bd-4691396ffafc\" (UID: \"0b124a84-203f-4628-96bd-4691396ffafc\") " Oct 30 19:08:48.008660 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.008649 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b124a84-203f-4628-96bd-4691396ffafc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b124a84-203f-4628-96bd-4691396ffafc-plugins-config-volume\") pod \"0b124a84-203f-4628-96bd-4691396ffafc\" (UID: \"0b124a84-203f-4628-96bd-4691396ffafc\") " Oct 30 19:08:48.008963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.008947 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54bvv\" (UniqueName: \"kubernetes.io/projected/fa302575-37dc-4227-a316-56258980eb26-kube-api-access-54bvv\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:48.009052 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.008974 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b124a84-203f-4628-96bd-4691396ffafc-plugins-config-volume" pod "0b124a84-203f-4628-96bd-4691396ffafc" (UID: "0b124a84-203f-4628-96bd-4691396ffafc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:48.010735 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.010708 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b124a84-203f-4628-96bd-4691396ffafc-kube-api-access-ztj74" pod "0b124a84-203f-4628-96bd-4691396ffafc" (UID: "0b124a84-203f-4628-96bd-4691396ffafc"). InnerVolumeSpecName "kube-api-access-ztj74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:48.047111 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.047080 4862 generic.go:356] "Generic (PLEG): container finished" podID="fa302575-37dc-4227-a316-56258980eb26" containerID="71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a" exitCode=0 Oct 30 19:08:48.047283 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.047138 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" Oct 30 19:08:48.047283 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.047166 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" event={"ID":"fa302575-37dc-4227-a316-56258980eb26","Type":"ContainerDied","Data":"71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a"} Oct 30 19:08:48.047283 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.047247 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb" event={"ID":"fa302575-37dc-4227-a316-56258980eb26","Type":"ContainerDied","Data":"5cdf93e20bda6b212c031aea7d20ba3326de5e2c45aabbbab6c64ca356c9a7f5"} Oct 30 19:08:48.047463 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.047301 4862 scope.go:122] "RemoveContainer" containerID="71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a" Oct 30 19:08:48.048868 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.048842 4862 generic.go:356] "Generic (PLEG): container finished" podID="0b124a84-203f-4628-96bd-4691396ffafc" containerID="5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd" exitCode=0 Oct 30 19:08:48.049002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.048879 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" event={"ID":"0b124a84-203f-4628-96bd-4691396ffafc","Type":"ContainerDied","Data":"5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd"} Oct 30 19:08:48.049002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.048941 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" event={"ID":"0b124a84-203f-4628-96bd-4691396ffafc","Type":"ContainerDied","Data":"d92405e7ad1759f89625c3961eebb49d103a46935abc2cb364e0638dba3606b5"} Oct 30 19:08:48.049002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.048894 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6" Oct 30 19:08:48.059252 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.059205 4862 scope.go:122] "RemoveContainer" containerID="71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a" Oct 30 19:08:48.059547 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:08:48.059521 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a\": container with ID starting with 71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a not found: ID does not exist" containerID="71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a" Oct 30 19:08:48.059646 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.059558 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a"} err="failed to get container status \"71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a\": rpc error: code = NotFound desc = could not find container \"71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a\": container with ID starting with 71bb2eb944257c65fc074313d1f9a44acdeb4c2135c3b3b7f4fb90b3656be93a not found: ID does not exist" Oct 30 19:08:48.059646 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.059580 4862 scope.go:122] "RemoveContainer" containerID="5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd" Oct 30 19:08:48.070092 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.070073 4862 scope.go:122] "RemoveContainer" containerID="5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd" Oct 30 19:08:48.070371 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:08:48.070353 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd\": container with ID starting with 5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd not found: ID does not exist" containerID="5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd" Oct 30 19:08:48.070427 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.070379 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd"} err="failed to get container status \"5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd\": rpc error: code = NotFound desc = could not find container \"5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd\": container with ID starting with 5c19bcab434b0844afb5381d2cbec9c31053239e9c3fa1e95c3d663f05c3a2bd not found: ID does not exist" Oct 30 19:08:48.094781 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.094727 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6"] Oct 30 19:08:48.098848 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.098829 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e8827b-55679df55-xq5t6"] Oct 30 19:08:48.110193 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.110167 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b124a84-203f-4628-96bd-4691396ffafc-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:48.110307 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.110193 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztj74\" (UniqueName: \"kubernetes.io/projected/0b124a84-203f-4628-96bd-4691396ffafc-kube-api-access-ztj74\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:48.126430 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.126408 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb"] Oct 30 19:08:48.151871 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:48.151834 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e8827b-787b47bbf6-tclcb"] Oct 30 19:08:49.064705 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:49.064671 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b124a84-203f-4628-96bd-4691396ffafc" path="/var/lib/kubelet/pods/0b124a84-203f-4628-96bd-4691396ffafc/volumes" Oct 30 19:08:49.065294 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:49.065255 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa302575-37dc-4227-a316-56258980eb26" path="/var/lib/kubelet/pods/fa302575-37dc-4227-a316-56258980eb26/volumes" Oct 30 19:08:49.871122 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:08:49.871080 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:08:50.452196 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:50.452141 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 19:08:50.537275 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:50.537235 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3a2fa1-7a97-488b-8420-e9074e752891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a2fa1-7a97-488b-8420-e9074e752891-plugins-config-volume\") pod \"2c3a2fa1-7a97-488b-8420-e9074e752891\" (UID: \"2c3a2fa1-7a97-488b-8420-e9074e752891\") " Oct 30 19:08:50.537431 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:50.537399 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3a2fa1-7a97-488b-8420-e9074e752891-kube-api-access-g4nwg\" (UniqueName: \"kubernetes.io/projected/2c3a2fa1-7a97-488b-8420-e9074e752891-kube-api-access-g4nwg\") pod \"2c3a2fa1-7a97-488b-8420-e9074e752891\" (UID: \"2c3a2fa1-7a97-488b-8420-e9074e752891\") " Oct 30 19:08:50.537645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:50.537620 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3a2fa1-7a97-488b-8420-e9074e752891-plugins-config-volume" pod "2c3a2fa1-7a97-488b-8420-e9074e752891" (UID: "2c3a2fa1-7a97-488b-8420-e9074e752891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:50.537928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:50.537911 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c3a2fa1-7a97-488b-8420-e9074e752891-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:50.539688 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:50.539670 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3a2fa1-7a97-488b-8420-e9074e752891-kube-api-access-g4nwg" pod "2c3a2fa1-7a97-488b-8420-e9074e752891" (UID: "2c3a2fa1-7a97-488b-8420-e9074e752891"). InnerVolumeSpecName "kube-api-access-g4nwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:50.639720 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:50.639693 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4nwg\" (UniqueName: \"kubernetes.io/projected/2c3a2fa1-7a97-488b-8420-e9074e752891-kube-api-access-g4nwg\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:51.087994 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:51.087961 4862 generic.go:356] "Generic (PLEG): container finished" podID="2c3a2fa1-7a97-488b-8420-e9074e752891" containerID="385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8" exitCode=0 Oct 30 19:08:51.088140 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:51.088044 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" event={"ID":"2c3a2fa1-7a97-488b-8420-e9074e752891","Type":"ContainerDied","Data":"385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8"} Oct 30 19:08:51.088140 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:51.088096 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" event={"ID":"2c3a2fa1-7a97-488b-8420-e9074e752891","Type":"ContainerDied","Data":"de8c4c11e8ab81d523615b8cb78bb593b3973ff0be072a42eb66fa8294bee2a8"} Oct 30 19:08:51.088140 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:51.088123 4862 scope.go:122] "RemoveContainer" containerID="385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8" Oct 30 19:08:51.088348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:51.088330 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd" Oct 30 19:08:51.102356 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:51.102329 4862 scope.go:122] "RemoveContainer" containerID="385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8" Oct 30 19:08:51.102656 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:08:51.102635 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8\": container with ID starting with 385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8 not found: ID does not exist" containerID="385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8" Oct 30 19:08:51.102750 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:51.102668 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8"} err="failed to get container status \"385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8\": rpc error: code = NotFound desc = could not find container \"385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8\": container with ID starting with 385d2d563d2b59e69eb80a6ac20437e85b059f9c70d79c13ab36319a38982ad8 not found: ID does not exist" Oct 30 19:08:51.134931 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:51.134892 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd"] Oct 30 19:08:51.144932 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:51.144907 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-38e0db-674844768b-xkbkd"] Oct 30 19:08:53.062366 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:53.062332 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3a2fa1-7a97-488b-8420-e9074e752891" path="/var/lib/kubelet/pods/2c3a2fa1-7a97-488b-8420-e9074e752891/volumes" Oct 30 19:08:54.459477 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.459437 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z"] Oct 30 19:08:54.460006 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.459993 4862 cpu_manager.go:414] "containerMap: removing container" podUID="2c3a2fa1-7a97-488b-8420-e9074e752891" containerName="payload-processing" Oct 30 19:08:54.460056 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460007 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3a2fa1-7a97-488b-8420-e9074e752891" containerName="payload-processing" Oct 30 19:08:54.460056 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460020 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0b124a84-203f-4628-96bd-4691396ffafc" containerName="payload-pre-processing" Oct 30 19:08:54.460056 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460025 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b124a84-203f-4628-96bd-4691396ffafc" containerName="payload-pre-processing" Oct 30 19:08:54.460056 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460049 4862 cpu_manager.go:414] "containerMap: removing container" podUID="fa302575-37dc-4227-a316-56258980eb26" containerName="payload-processing" Oct 30 19:08:54.460056 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460054 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa302575-37dc-4227-a316-56258980eb26" containerName="payload-processing" Oct 30 19:08:54.460224 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460063 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6a753f62-84e2-4ba1-a32b-f46093e2f672" containerName="revoke-keys" Oct 30 19:08:54.460224 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460067 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a753f62-84e2-4ba1-a32b-f46093e2f672" containerName="revoke-keys" Oct 30 19:08:54.460224 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460159 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa302575-37dc-4227-a316-56258980eb26" containerName="payload-processing" Oct 30 19:08:54.460224 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460169 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a753f62-84e2-4ba1-a32b-f46093e2f672" containerName="revoke-keys" Oct 30 19:08:54.460224 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460180 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3a2fa1-7a97-488b-8420-e9074e752891" containerName="payload-processing" Oct 30 19:08:54.460224 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.460186 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b124a84-203f-4628-96bd-4691396ffafc" containerName="payload-pre-processing" Oct 30 19:08:54.464849 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.464830 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.481979 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.481958 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z"] Oct 30 19:08:54.545521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.545494 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.545650 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.545614 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9js2\" (UniqueName: \"kubernetes.io/projected/5e906db6-7868-4fbb-9f76-10193eaacb45-kube-api-access-r9js2\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.545721 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.545705 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.566293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.566254 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:08:54.566425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.566311 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:08:54.647164 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.647132 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.647336 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.647199 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9js2\" (UniqueName: \"kubernetes.io/projected/5e906db6-7868-4fbb-9f76-10193eaacb45-kube-api-access-r9js2\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.647336 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.647288 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.647922 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.647849 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.649699 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.649677 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.665178 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.665155 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9js2\" (UniqueName: \"kubernetes.io/projected/5e906db6-7868-4fbb-9f76-10193eaacb45-kube-api-access-r9js2\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:54.778920 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:54.778861 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:55.083743 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:55.083719 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z"] Oct 30 19:08:55.085913 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:08:55.085891 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e906db6_7868_4fbb_9f76_10193eaacb45.slice/crio-288e30ce1236227f924591b1e7c8c422d618e50c99da589757d834690112906e WatchSource:0}: Error finding container 288e30ce1236227f924591b1e7c8c422d618e50c99da589757d834690112906e: Status 404 returned error can't find the container with id 288e30ce1236227f924591b1e7c8c422d618e50c99da589757d834690112906e Oct 30 19:08:55.118096 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:55.118073 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" event={"ID":"5e906db6-7868-4fbb-9f76-10193eaacb45","Type":"ContainerStarted","Data":"288e30ce1236227f924591b1e7c8c422d618e50c99da589757d834690112906e"} Oct 30 19:08:56.127443 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:56.127237 4862 generic.go:356] "Generic (PLEG): container finished" podID="5e906db6-7868-4fbb-9f76-10193eaacb45" containerID="a2aeb6f9663f779a3eec7d37370fd4fc3b8f601060ce8b523a8bbb16996e7c3c" exitCode=0 Oct 30 19:08:56.127443 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:56.127335 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" event={"ID":"5e906db6-7868-4fbb-9f76-10193eaacb45","Type":"ContainerDied","Data":"a2aeb6f9663f779a3eec7d37370fd4fc3b8f601060ce8b523a8bbb16996e7c3c"} Oct 30 19:08:57.297578 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.297545 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:57.379163 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.379128 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e906db6-7868-4fbb-9f76-10193eaacb45-kube-api-access-r9js2\" (UniqueName: \"kubernetes.io/projected/5e906db6-7868-4fbb-9f76-10193eaacb45-kube-api-access-r9js2\") pod \"5e906db6-7868-4fbb-9f76-10193eaacb45\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " Oct 30 19:08:57.379354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.379245 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-openshift-ca\") pod \"5e906db6-7868-4fbb-9f76-10193eaacb45\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " Oct 30 19:08:57.379436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.379413 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-xks-ca\") pod \"5e906db6-7868-4fbb-9f76-10193eaacb45\" (UID: \"5e906db6-7868-4fbb-9f76-10193eaacb45\") " Oct 30 19:08:57.379635 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.379604 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-openshift-ca" pod "5e906db6-7868-4fbb-9f76-10193eaacb45" (UID: "5e906db6-7868-4fbb-9f76-10193eaacb45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:57.379979 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.379960 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:57.381648 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.381622 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-xks-ca" pod "5e906db6-7868-4fbb-9f76-10193eaacb45" (UID: "5e906db6-7868-4fbb-9f76-10193eaacb45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:08:57.381648 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.381631 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e906db6-7868-4fbb-9f76-10193eaacb45-kube-api-access-r9js2" pod "5e906db6-7868-4fbb-9f76-10193eaacb45" (UID: "5e906db6-7868-4fbb-9f76-10193eaacb45"). InnerVolumeSpecName "kube-api-access-r9js2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:57.481467 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.481445 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9js2\" (UniqueName: \"kubernetes.io/projected/5e906db6-7868-4fbb-9f76-10193eaacb45-kube-api-access-r9js2\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:57.481467 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:57.481467 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e906db6-7868-4fbb-9f76-10193eaacb45-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:08:58.141475 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:58.141448 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" Oct 30 19:08:58.141642 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:58.141494 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z" event={"ID":"5e906db6-7868-4fbb-9f76-10193eaacb45","Type":"ContainerDied","Data":"288e30ce1236227f924591b1e7c8c422d618e50c99da589757d834690112906e"} Oct 30 19:08:58.141642 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:08:58.141551 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288e30ce1236227f924591b1e7c8c422d618e50c99da589757d834690112906e" Oct 30 19:09:13.872154 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:13.872121 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n"] Oct 30 19:09:13.881456 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:13.881430 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e8827b-rcf9n"] Oct 30 19:09:14.028955 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:14.028923 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l"] Oct 30 19:09:14.031989 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:14.031956 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9411391-7gv2l"] Oct 30 19:09:15.063650 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:15.063615 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2a0513-e682-4a1e-aa20-c6f3e310ffab" path="/var/lib/kubelet/pods/5f2a0513-e682-4a1e-aa20-c6f3e310ffab/volumes" Oct 30 19:09:15.064075 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:15.064064 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9b84d3-de21-4ff9-b515-02a7cdb79557" path="/var/lib/kubelet/pods/ac9b84d3-de21-4ff9-b515-02a7cdb79557/volumes" Oct 30 19:09:19.873073 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:09:19.873032 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:09:23.484881 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:23.484832 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5"] Oct 30 19:09:23.490912 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:23.490126 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ab244c-dx9p5"] Oct 30 19:09:25.062687 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:25.062652 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a753f62-84e2-4ba1-a32b-f46093e2f672" path="/var/lib/kubelet/pods/6a753f62-84e2-4ba1-a32b-f46093e2f672/volumes" Oct 30 19:09:25.855887 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:25.855846 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t"] Oct 30 19:09:25.856441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:25.856424 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5e906db6-7868-4fbb-9f76-10193eaacb45" containerName="revoke-keys" Oct 30 19:09:25.856441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:25.856441 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e906db6-7868-4fbb-9f76-10193eaacb45" containerName="revoke-keys" Oct 30 19:09:25.856616 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:25.856536 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e906db6-7868-4fbb-9f76-10193eaacb45" containerName="revoke-keys" Oct 30 19:09:25.861681 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:25.861658 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:25.876680 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:25.876663 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-24b124-openshift-default-dockercfg-qbtjb\"" Oct 30 19:09:25.890796 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:25.890769 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t"] Oct 30 19:09:26.035145 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.035110 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-token\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.035348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.035303 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.035348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.035335 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-data\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.035505 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.035482 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v9cx\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-kube-api-access-8v9cx\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.035563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.035521 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4271c0fb-4068-4a80-a095-418dad3b8c60-istiod-ca-cert\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.035717 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.035697 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-certs\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.035768 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.035749 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-envoy\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.035840 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.035823 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-podinfo\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.035908 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.035870 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-credential-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.136562 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.136496 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-envoy\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.136562 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.136536 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-podinfo\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.136562 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.136562 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-credential-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.136595 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-token\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.136657 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.136674 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-data\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.136730 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v9cx\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-kube-api-access-8v9cx\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.136749 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4271c0fb-4068-4a80-a095-418dad3b8c60-istiod-ca-cert\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.136833 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-certs\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.137121 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-credential-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.137166 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-data\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.137203 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-certs\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.137343 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-socket\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.137837 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.137813 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4271c0fb-4068-4a80-a095-418dad3b8c60-istiod-ca-cert\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.138958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.138932 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-envoy\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.139353 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.139336 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-podinfo\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.154483 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.154456 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-token\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.154638 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.154497 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v9cx\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-kube-api-access-8v9cx\") pod \"e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.178767 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.178745 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:26.503036 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.503003 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t"] Oct 30 19:09:26.508304 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:09:26.506049 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4271c0fb_4068_4a80_a095_418dad3b8c60.slice/crio-70239678e70e59b69b4e365b19807d8a6af04e28f820bf999e00a6609cd41d6f WatchSource:0}: Error finding container 70239678e70e59b69b4e365b19807d8a6af04e28f820bf999e00a6609cd41d6f: Status 404 returned error can't find the container with id 70239678e70e59b69b4e365b19807d8a6af04e28f820bf999e00a6609cd41d6f Oct 30 19:09:26.509307 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.509236 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:09:26.509408 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.509351 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:09:26.509485 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:26.509417 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:09:27.386773 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:27.386742 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" event={"ID":"4271c0fb-4068-4a80-a095-418dad3b8c60","Type":"ContainerStarted","Data":"3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec"} Oct 30 19:09:27.387318 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:27.386790 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" event={"ID":"4271c0fb-4068-4a80-a095-418dad3b8c60","Type":"ContainerStarted","Data":"70239678e70e59b69b4e365b19807d8a6af04e28f820bf999e00a6609cd41d6f"} Oct 30 19:09:27.453883 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:27.453813 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" podStartSLOduration=2.453793311 podStartE2EDuration="2.453793311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:27.453349555 +0000 UTC m=+4266.502394477" watchObservedRunningTime="2026-10-30 19:09:27.453793311 +0000 UTC m=+4266.502838227" Oct 30 19:09:28.179190 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:28.179158 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:28.181067 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:28.181042 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:09:28.181223 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:28.181087 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 19:09:29.179318 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:29.179258 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:09:29.179721 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:29.179346 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 19:09:30.179674 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:30.179635 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:09:30.180071 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:30.179695 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 19:09:31.192338 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:31.188665 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:09:31.192338 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:31.188730 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 19:09:32.179552 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.179517 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 19:09:32.179742 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.179584 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 19:09:32.641298 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.641238 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg"] Oct 30 19:09:32.646521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.646499 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.677701 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.675924 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:32.677701 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.676141 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:09:32.677701 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.676823 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg"] Oct 30 19:09:32.778253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.778214 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsr9x\" (UniqueName: \"kubernetes.io/projected/f9e330dc-6b6b-4a18-ab80-1979b576b17a-kube-api-access-jsr9x\") pod \"maas-api-revoke-sub-05a205a9-cdvmg\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.778253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.778257 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05a205a9-cdvmg\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.778560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.778374 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05a205a9-cdvmg\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.879380 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.879350 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsr9x\" (UniqueName: \"kubernetes.io/projected/f9e330dc-6b6b-4a18-ab80-1979b576b17a-kube-api-access-jsr9x\") pod \"maas-api-revoke-sub-05a205a9-cdvmg\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.879584 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.879394 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05a205a9-cdvmg\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.879584 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.879425 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05a205a9-cdvmg\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.880244 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.880219 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05a205a9-cdvmg\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.882334 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.882306 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05a205a9-cdvmg\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.897874 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.897818 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsr9x\" (UniqueName: \"kubernetes.io/projected/f9e330dc-6b6b-4a18-ab80-1979b576b17a-kube-api-access-jsr9x\") pod \"maas-api-revoke-sub-05a205a9-cdvmg\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:32.983432 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:32.983403 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:33.184254 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:33.184179 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:33.184745 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:33.184726 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:33.185547 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:33.185526 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:09:33.303525 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:33.303495 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg"] Oct 30 19:09:33.306348 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:09:33.306320 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e330dc_6b6b_4a18_ab80_1979b576b17a.slice/crio-ef2408e765305c36bd93e8cc77135250d940abb2dfa4d3f772288bf750129fe9 WatchSource:0}: Error finding container ef2408e765305c36bd93e8cc77135250d940abb2dfa4d3f772288bf750129fe9: Status 404 returned error can't find the container with id ef2408e765305c36bd93e8cc77135250d940abb2dfa4d3f772288bf750129fe9 Oct 30 19:09:33.434075 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:33.434040 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" event={"ID":"f9e330dc-6b6b-4a18-ab80-1979b576b17a","Type":"ContainerStarted","Data":"ef2408e765305c36bd93e8cc77135250d940abb2dfa4d3f772288bf750129fe9"} Oct 30 19:09:34.444024 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:34.443932 4862 generic.go:356] "Generic (PLEG): container finished" podID="f9e330dc-6b6b-4a18-ab80-1979b576b17a" containerID="bd83d93ff6f4dec0b917389dc1a55521b35d05f9f944b4e0c6e002668cfc3b41" exitCode=0 Oct 30 19:09:34.444583 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:34.444029 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" event={"ID":"f9e330dc-6b6b-4a18-ab80-1979b576b17a","Type":"ContainerDied","Data":"bd83d93ff6f4dec0b917389dc1a55521b35d05f9f944b4e0c6e002668cfc3b41"} Oct 30 19:09:35.026712 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.026672 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv"] Oct 30 19:09:35.037090 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.037062 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003418fc-qvjbv"] Oct 30 19:09:35.063007 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.062977 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cceaf4e-429c-41a3-90f6-4f575a662229" path="/var/lib/kubelet/pods/3cceaf4e-429c-41a3-90f6-4f575a662229/volumes" Oct 30 19:09:35.624451 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.624424 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:35.718527 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.718441 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-xks-ca\") pod \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " Oct 30 19:09:35.718693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.718610 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9e330dc-6b6b-4a18-ab80-1979b576b17a-kube-api-access-jsr9x\" (UniqueName: \"kubernetes.io/projected/f9e330dc-6b6b-4a18-ab80-1979b576b17a-kube-api-access-jsr9x\") pod \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " Oct 30 19:09:35.718804 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.718789 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-openshift-ca\") pod \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\" (UID: \"f9e330dc-6b6b-4a18-ab80-1979b576b17a\") " Oct 30 19:09:35.719124 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.719096 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-openshift-ca" pod "f9e330dc-6b6b-4a18-ab80-1979b576b17a" (UID: "f9e330dc-6b6b-4a18-ab80-1979b576b17a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:35.720689 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.720665 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-xks-ca" pod "f9e330dc-6b6b-4a18-ab80-1979b576b17a" (UID: "f9e330dc-6b6b-4a18-ab80-1979b576b17a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:35.720903 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.720882 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9e330dc-6b6b-4a18-ab80-1979b576b17a-kube-api-access-jsr9x" pod "f9e330dc-6b6b-4a18-ab80-1979b576b17a" (UID: "f9e330dc-6b6b-4a18-ab80-1979b576b17a"). InnerVolumeSpecName "kube-api-access-jsr9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:35.820086 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.820055 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsr9x\" (UniqueName: \"kubernetes.io/projected/f9e330dc-6b6b-4a18-ab80-1979b576b17a-kube-api-access-jsr9x\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:35.820086 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.820084 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:35.820297 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:35.820094 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9e330dc-6b6b-4a18-ab80-1979b576b17a-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:36.460012 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:36.459984 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" Oct 30 19:09:36.460195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:36.460027 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg" event={"ID":"f9e330dc-6b6b-4a18-ab80-1979b576b17a","Type":"ContainerDied","Data":"ef2408e765305c36bd93e8cc77135250d940abb2dfa4d3f772288bf750129fe9"} Oct 30 19:09:36.460195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:36.460079 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef2408e765305c36bd93e8cc77135250d940abb2dfa4d3f772288bf750129fe9" Oct 30 19:09:36.811293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:36.811221 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z"] Oct 30 19:09:36.814467 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:36.814445 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d82e82-rh26z"] Oct 30 19:09:37.062467 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:37.062400 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e906db6-7868-4fbb-9f76-10193eaacb45" path="/var/lib/kubelet/pods/5e906db6-7868-4fbb-9f76-10193eaacb45/volumes" Oct 30 19:09:40.030891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:40.030857 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf"] Oct 30 19:09:40.040991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:40.040967 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-112a46ec-vbkvf"] Oct 30 19:09:40.380478 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:40.380438 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:09:41.079698 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:41.079658 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8950842-58b0-4ac1-a6e4-780b719111fb" path="/var/lib/kubelet/pods/d8950842-58b0-4ac1-a6e4-780b719111fb/volumes" Oct 30 19:09:44.692105 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:44.692070 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv"] Oct 30 19:09:44.692644 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:44.692298 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" containerID="cri-o://df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796" gracePeriod=30 Oct 30 19:09:45.391436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.391398 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk"] Oct 30 19:09:45.392027 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.392006 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f9e330dc-6b6b-4a18-ab80-1979b576b17a" containerName="revoke-keys" Oct 30 19:09:45.392216 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.392028 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9e330dc-6b6b-4a18-ab80-1979b576b17a" containerName="revoke-keys" Oct 30 19:09:45.392310 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.392227 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9e330dc-6b6b-4a18-ab80-1979b576b17a" containerName="revoke-keys" Oct 30 19:09:45.397154 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.397132 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.417365 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.417342 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk"] Oct 30 19:09:45.423504 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.423482 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:45.423640 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.423622 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:09:45.465816 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.465785 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.466017 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.465969 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.466082 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.466023 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn4vp\" (UniqueName: \"kubernetes.io/projected/c4d775d3-579f-4166-8929-165b759b8df2-kube-api-access-xn4vp\") pod \"maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.567300 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.567251 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.567508 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.567325 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn4vp\" (UniqueName: \"kubernetes.io/projected/c4d775d3-579f-4166-8929-165b759b8df2-kube-api-access-xn4vp\") pod \"maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.567508 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.567464 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.568178 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.568156 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.569903 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.569882 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.586846 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.586823 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn4vp\" (UniqueName: \"kubernetes.io/projected/c4d775d3-579f-4166-8929-165b759b8df2-kube-api-access-xn4vp\") pod \"maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:45.714383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:45.714296 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:46.036534 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:46.036510 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk"] Oct 30 19:09:46.040412 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:09:46.040381 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d775d3_579f_4166_8929_165b759b8df2.slice/crio-eb0a9f5277f9f01ad3636276d73a41837ae64208fc77e76a0309305e79ec10d7 WatchSource:0}: Error finding container eb0a9f5277f9f01ad3636276d73a41837ae64208fc77e76a0309305e79ec10d7: Status 404 returned error can't find the container with id eb0a9f5277f9f01ad3636276d73a41837ae64208fc77e76a0309305e79ec10d7 Oct 30 19:09:46.533333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:46.533296 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" event={"ID":"c4d775d3-579f-4166-8929-165b759b8df2","Type":"ContainerStarted","Data":"63e9274bc986f5f12a92edcfb1cd946ab29ac0e40388668ccd4bba12de6c8274"} Oct 30 19:09:46.533495 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:46.533363 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" event={"ID":"c4d775d3-579f-4166-8929-165b759b8df2","Type":"ContainerStarted","Data":"eb0a9f5277f9f01ad3636276d73a41837ae64208fc77e76a0309305e79ec10d7"} Oct 30 19:09:46.568423 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:46.568377 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" podStartSLOduration=1.568363042 podStartE2EDuration="1.568363042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:46.56771927 +0000 UTC m=+4285.616764195" watchObservedRunningTime="2026-10-30 19:09:46.568363042 +0000 UTC m=+4285.617407947" Oct 30 19:09:47.541956 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:47.541922 4862 generic.go:356] "Generic (PLEG): container finished" podID="c4d775d3-579f-4166-8929-165b759b8df2" containerID="63e9274bc986f5f12a92edcfb1cd946ab29ac0e40388668ccd4bba12de6c8274" exitCode=0 Oct 30 19:09:47.542447 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:47.541973 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" event={"ID":"c4d775d3-579f-4166-8929-165b759b8df2","Type":"ContainerDied","Data":"63e9274bc986f5f12a92edcfb1cd946ab29ac0e40388668ccd4bba12de6c8274"} Oct 30 19:09:48.763615 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.763589 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:48.808541 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.808456 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-xks-ca\") pod \"c4d775d3-579f-4166-8929-165b759b8df2\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " Oct 30 19:09:48.808695 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.808628 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-openshift-ca\") pod \"c4d775d3-579f-4166-8929-165b759b8df2\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " Oct 30 19:09:48.808793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.808775 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d775d3-579f-4166-8929-165b759b8df2-kube-api-access-xn4vp\" (UniqueName: \"kubernetes.io/projected/c4d775d3-579f-4166-8929-165b759b8df2-kube-api-access-xn4vp\") pod \"c4d775d3-579f-4166-8929-165b759b8df2\" (UID: \"c4d775d3-579f-4166-8929-165b759b8df2\") " Oct 30 19:09:48.809791 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.809742 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-openshift-ca" pod "c4d775d3-579f-4166-8929-165b759b8df2" (UID: "c4d775d3-579f-4166-8929-165b759b8df2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:48.810330 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.810310 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:48.811860 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.811840 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-xks-ca" pod "c4d775d3-579f-4166-8929-165b759b8df2" (UID: "c4d775d3-579f-4166-8929-165b759b8df2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:09:48.811978 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.811878 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d775d3-579f-4166-8929-165b759b8df2-kube-api-access-xn4vp" pod "c4d775d3-579f-4166-8929-165b759b8df2" (UID: "c4d775d3-579f-4166-8929-165b759b8df2"). InnerVolumeSpecName "kube-api-access-xn4vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:48.911596 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.911568 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn4vp\" (UniqueName: \"kubernetes.io/projected/c4d775d3-579f-4166-8929-165b759b8df2-kube-api-access-xn4vp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:48.911596 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:48.911594 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d775d3-579f-4166-8929-165b759b8df2-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:49.564021 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:49.563980 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" event={"ID":"c4d775d3-579f-4166-8929-165b759b8df2","Type":"ContainerDied","Data":"eb0a9f5277f9f01ad3636276d73a41837ae64208fc77e76a0309305e79ec10d7"} Oct 30 19:09:49.564230 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:49.564067 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb0a9f5277f9f01ad3636276d73a41837ae64208fc77e76a0309305e79ec10d7" Oct 30 19:09:49.564230 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:49.563996 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk" Oct 30 19:09:49.664360 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:49.664326 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:09:49.664535 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:49.664385 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:09:49.872095 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:09:49.872001 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:09:49.987655 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:49.987632 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 19:09:50.034597 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.034570 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-kube-api-access-5xkw6\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-kube-api-access-5xkw6\") pod \"a4a7ed92-3470-4734-9896-3ce0def49be1\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " Oct 30 19:09:50.034771 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.034721 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-podinfo\") pod \"a4a7ed92-3470-4734-9896-3ce0def49be1\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " Oct 30 19:09:50.034771 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.034748 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-credential-socket\") pod \"a4a7ed92-3470-4734-9896-3ce0def49be1\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " Oct 30 19:09:50.034901 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.034808 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-token\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-token\") pod \"a4a7ed92-3470-4734-9896-3ce0def49be1\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " Oct 30 19:09:50.034901 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.034827 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a7ed92-3470-4734-9896-3ce0def49be1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a7ed92-3470-4734-9896-3ce0def49be1-istiod-ca-cert\") pod \"a4a7ed92-3470-4734-9896-3ce0def49be1\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " Oct 30 19:09:50.034901 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.034867 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-certs\") pod \"a4a7ed92-3470-4734-9896-3ce0def49be1\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " Oct 30 19:09:50.035072 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.034936 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-data\") pod \"a4a7ed92-3470-4734-9896-3ce0def49be1\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " Oct 30 19:09:50.035072 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.034954 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-socket\") pod \"a4a7ed92-3470-4734-9896-3ce0def49be1\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " Oct 30 19:09:50.035072 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.034984 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-envoy\") pod \"a4a7ed92-3470-4734-9896-3ce0def49be1\" (UID: \"a4a7ed92-3470-4734-9896-3ce0def49be1\") " Oct 30 19:09:50.035772 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.035612 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-credential-socket" pod "a4a7ed92-3470-4734-9896-3ce0def49be1" (UID: "a4a7ed92-3470-4734-9896-3ce0def49be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.035894 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.035821 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-data" pod "a4a7ed92-3470-4734-9896-3ce0def49be1" (UID: "a4a7ed92-3470-4734-9896-3ce0def49be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.035951 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.035923 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-certs" pod "a4a7ed92-3470-4734-9896-3ce0def49be1" (UID: "a4a7ed92-3470-4734-9896-3ce0def49be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.035951 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.035939 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-socket" pod "a4a7ed92-3470-4734-9896-3ce0def49be1" (UID: "a4a7ed92-3470-4734-9896-3ce0def49be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.036114 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.036089 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a7ed92-3470-4734-9896-3ce0def49be1-istiod-ca-cert" pod "a4a7ed92-3470-4734-9896-3ce0def49be1" (UID: "a4a7ed92-3470-4734-9896-3ce0def49be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:50.038407 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.038380 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-envoy" pod "a4a7ed92-3470-4734-9896-3ce0def49be1" (UID: "a4a7ed92-3470-4734-9896-3ce0def49be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:09:50.038529 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.038469 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-podinfo" pod "a4a7ed92-3470-4734-9896-3ce0def49be1" (UID: "a4a7ed92-3470-4734-9896-3ce0def49be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:09:50.043002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.039199 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-kube-api-access-5xkw6" pod "a4a7ed92-3470-4734-9896-3ce0def49be1" (UID: "a4a7ed92-3470-4734-9896-3ce0def49be1"). InnerVolumeSpecName "kube-api-access-5xkw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:50.043002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.042674 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-token" pod "a4a7ed92-3470-4734-9896-3ce0def49be1" (UID: "a4a7ed92-3470-4734-9896-3ce0def49be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:50.133901 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.133818 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx"] Oct 30 19:09:50.136688 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.136660 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:50.136825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.136693 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:50.136825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.136706 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:50.136825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.136720 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a7ed92-3470-4734-9896-3ce0def49be1-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:50.136825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.136732 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:50.136825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.136744 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:50.136825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.136757 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:50.136825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.136771 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a7ed92-3470-4734-9896-3ce0def49be1-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:50.136825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.136784 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xkw6\" (UniqueName: \"kubernetes.io/projected/a4a7ed92-3470-4734-9896-3ce0def49be1-kube-api-access-5xkw6\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:09:50.143731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.143704 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq"] Oct 30 19:09:50.576553 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.576516 4862 generic.go:356] "Generic (PLEG): container finished" podID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerID="df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796" exitCode=0 Oct 30 19:09:50.576761 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.576573 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" Oct 30 19:09:50.576761 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.576596 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" event={"ID":"a4a7ed92-3470-4734-9896-3ce0def49be1","Type":"ContainerDied","Data":"df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796"} Oct 30 19:09:50.576761 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.576664 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv" event={"ID":"a4a7ed92-3470-4734-9896-3ce0def49be1","Type":"ContainerDied","Data":"1e722f90ffd167c6c24c01e595872e1be2db3d84e44097c6d1931a6653ea7c02"} Oct 30 19:09:50.576761 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.576691 4862 scope.go:122] "RemoveContainer" containerID="df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796" Oct 30 19:09:50.588647 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.588631 4862 scope.go:122] "RemoveContainer" containerID="b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d" Oct 30 19:09:50.598338 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.598323 4862 scope.go:122] "RemoveContainer" containerID="df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796" Oct 30 19:09:50.598596 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:09:50.598570 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796\": container with ID starting with df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796 not found: ID does not exist" containerID="df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796" Oct 30 19:09:50.598697 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.598601 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796"} err="failed to get container status \"df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796\": rpc error: code = NotFound desc = could not find container \"df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796\": container with ID starting with df1dfd3997fd623cf88c8ec9a6cdec64e18e51e52ef81b962761c0c3960be796 not found: ID does not exist" Oct 30 19:09:50.598697 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.598616 4862 scope.go:122] "RemoveContainer" containerID="b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d" Oct 30 19:09:50.598859 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:09:50.598845 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d\": container with ID starting with b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d not found: ID does not exist" containerID="b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d" Oct 30 19:09:50.598903 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.598861 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d"} err="failed to get container status \"b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d\": rpc error: code = NotFound desc = could not find container \"b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d\": container with ID starting with b8eb629ca5d106f04e372328480afb9c650c89f5aa40eea685c94c24a8fcf62d not found: ID does not exist" Oct 30 19:09:50.624063 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.624041 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv"] Oct 30 19:09:50.628034 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:50.628013 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-d82e82-openshift-default-58686789bf-djgfv"] Oct 30 19:09:51.062894 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:51.062865 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" path="/var/lib/kubelet/pods/a4a7ed92-3470-4734-9896-3ce0def49be1/volumes" Oct 30 19:09:55.028106 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:55.028076 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b"] Oct 30 19:09:55.037910 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:55.037886 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b60ae622-gsl5b"] Oct 30 19:09:55.062456 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:55.062429 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ce0962-1b98-4895-9fb2-c2377e33568d" path="/var/lib/kubelet/pods/c2ce0962-1b98-4895-9fb2-c2377e33568d/volumes" Oct 30 19:09:59.031349 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.031312 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f"] Oct 30 19:09:59.041792 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.041760 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9190fff-fbd6f"] Oct 30 19:09:59.063068 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.063039 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8590f4d0-5a3c-4058-b66c-2d1dcce6222b" path="/var/lib/kubelet/pods/8590f4d0-5a3c-4058-b66c-2d1dcce6222b/volumes" Oct 30 19:09:59.624012 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.623978 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725"] Oct 30 19:09:59.624625 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.624601 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c4d775d3-579f-4166-8929-165b759b8df2" containerName="revoke-keys" Oct 30 19:09:59.624625 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.624623 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d775d3-579f-4166-8929-165b759b8df2" containerName="revoke-keys" Oct 30 19:09:59.624862 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.624651 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" Oct 30 19:09:59.624862 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.624657 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" Oct 30 19:09:59.624862 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.624695 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" Oct 30 19:09:59.624862 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.624704 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" Oct 30 19:09:59.624862 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.624831 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" Oct 30 19:09:59.624862 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.624847 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a7ed92-3470-4734-9896-3ce0def49be1" containerName="istio-proxy" Oct 30 19:09:59.624862 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.624858 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d775d3-579f-4166-8929-165b759b8df2" containerName="revoke-keys" Oct 30 19:09:59.628295 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.628255 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.647227 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.647193 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725"] Oct 30 19:09:59.716486 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.716455 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:09:59.716615 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.716500 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:09:59.784700 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.784671 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c318003-84725\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.784840 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.784732 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7mk\" (UniqueName: \"kubernetes.io/projected/5cc43bf9-1f43-473c-be5b-7130e798afd3-kube-api-access-ll7mk\") pod \"maas-api-revoke-sub-4c318003-84725\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.784957 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.784937 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c318003-84725\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.886372 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.886310 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7mk\" (UniqueName: \"kubernetes.io/projected/5cc43bf9-1f43-473c-be5b-7130e798afd3-kube-api-access-ll7mk\") pod \"maas-api-revoke-sub-4c318003-84725\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.886497 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.886400 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c318003-84725\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.886497 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.886477 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c318003-84725\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.887061 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.887038 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c318003-84725\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.888946 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.888918 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c318003-84725\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.907235 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.907208 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7mk\" (UniqueName: \"kubernetes.io/projected/5cc43bf9-1f43-473c-be5b-7130e798afd3-kube-api-access-ll7mk\") pod \"maas-api-revoke-sub-4c318003-84725\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:09:59.942415 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:09:59.942394 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:10:00.249369 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:00.249297 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725"] Oct 30 19:10:00.252781 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:10:00.252743 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc43bf9_1f43_473c_be5b_7130e798afd3.slice/crio-701e3971a2a7f5c5719a3257f0f0eca6f3bce940908f5de2a4f789ece715ac35 WatchSource:0}: Error finding container 701e3971a2a7f5c5719a3257f0f0eca6f3bce940908f5de2a4f789ece715ac35: Status 404 returned error can't find the container with id 701e3971a2a7f5c5719a3257f0f0eca6f3bce940908f5de2a4f789ece715ac35 Oct 30 19:10:00.663949 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:00.663903 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" event={"ID":"5cc43bf9-1f43-473c-be5b-7130e798afd3","Type":"ContainerStarted","Data":"6e81277e82de9c095b38973252f37566bd508e23ee312342dc9add3bdf65bd32"} Oct 30 19:10:00.664130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:00.663971 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" event={"ID":"5cc43bf9-1f43-473c-be5b-7130e798afd3","Type":"ContainerStarted","Data":"701e3971a2a7f5c5719a3257f0f0eca6f3bce940908f5de2a4f789ece715ac35"} Oct 30 19:10:00.702593 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:00.702541 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" podStartSLOduration=1.702527541 podStartE2EDuration="1.702527541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:00.699050854 +0000 UTC m=+4299.748095765" watchObservedRunningTime="2026-10-30 19:10:00.702527541 +0000 UTC m=+4299.751572455" Oct 30 19:10:01.672852 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:01.672814 4862 generic.go:356] "Generic (PLEG): container finished" podID="5cc43bf9-1f43-473c-be5b-7130e798afd3" containerID="6e81277e82de9c095b38973252f37566bd508e23ee312342dc9add3bdf65bd32" exitCode=0 Oct 30 19:10:01.673310 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:01.672901 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" event={"ID":"5cc43bf9-1f43-473c-be5b-7130e798afd3","Type":"ContainerDied","Data":"6e81277e82de9c095b38973252f37566bd508e23ee312342dc9add3bdf65bd32"} Oct 30 19:10:02.842498 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:02.842474 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:10:02.924839 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:02.924806 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-openshift-ca\") pod \"5cc43bf9-1f43-473c-be5b-7130e798afd3\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " Oct 30 19:10:02.925138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:02.925115 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc43bf9-1f43-473c-be5b-7130e798afd3-kube-api-access-ll7mk\" (UniqueName: \"kubernetes.io/projected/5cc43bf9-1f43-473c-be5b-7130e798afd3-kube-api-access-ll7mk\") pod \"5cc43bf9-1f43-473c-be5b-7130e798afd3\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " Oct 30 19:10:02.925232 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:02.925169 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-xks-ca\") pod \"5cc43bf9-1f43-473c-be5b-7130e798afd3\" (UID: \"5cc43bf9-1f43-473c-be5b-7130e798afd3\") " Oct 30 19:10:02.925320 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:02.925171 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-openshift-ca" pod "5cc43bf9-1f43-473c-be5b-7130e798afd3" (UID: "5cc43bf9-1f43-473c-be5b-7130e798afd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:02.925642 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:02.925619 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:02.927489 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:02.927465 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-xks-ca" pod "5cc43bf9-1f43-473c-be5b-7130e798afd3" (UID: "5cc43bf9-1f43-473c-be5b-7130e798afd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:02.927577 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:02.927497 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc43bf9-1f43-473c-be5b-7130e798afd3-kube-api-access-ll7mk" pod "5cc43bf9-1f43-473c-be5b-7130e798afd3" (UID: "5cc43bf9-1f43-473c-be5b-7130e798afd3"). InnerVolumeSpecName "kube-api-access-ll7mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:03.026962 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:03.026935 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7mk\" (UniqueName: \"kubernetes.io/projected/5cc43bf9-1f43-473c-be5b-7130e798afd3-kube-api-access-ll7mk\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:03.026962 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:03.026961 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc43bf9-1f43-473c-be5b-7130e798afd3-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:03.688734 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:03.688704 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" Oct 30 19:10:03.688887 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:03.688736 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725" event={"ID":"5cc43bf9-1f43-473c-be5b-7130e798afd3","Type":"ContainerDied","Data":"701e3971a2a7f5c5719a3257f0f0eca6f3bce940908f5de2a4f789ece715ac35"} Oct 30 19:10:03.688887 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:03.688787 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="701e3971a2a7f5c5719a3257f0f0eca6f3bce940908f5de2a4f789ece715ac35" Oct 30 19:10:05.156765 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.156729 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" containerID="cri-o://0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82" gracePeriod=45 Oct 30 19:10:05.164617 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.164557 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" containerID="cri-o://d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2" gracePeriod=45 Oct 30 19:10:05.547927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.547898 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 19:10:05.556416 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.553527 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.587813 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tmp-dir\") pod \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.587884 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tls-certs\") pod \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.587921 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8673e2ff-31e6-4172-ab13-8e5ea74713cf-kube-api-access-5cg4b\" (UniqueName: \"kubernetes.io/projected/8673e2ff-31e6-4172-ab13-8e5ea74713cf-kube-api-access-5cg4b\") pod \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.588021 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-model-cache\") pod \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.588074 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-dshm\") pod \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.588153 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tmp-dir\") pod \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.588202 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-dshm\") pod \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.588231 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-home\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-home\") pod \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\" (UID: \"8673e2ff-31e6-4172-ab13-8e5ea74713cf\") " Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.588332 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40fd49b0-06e5-4b9a-9489-ee3682f37a09-kube-api-access-bclfk\" (UniqueName: \"kubernetes.io/projected/40fd49b0-06e5-4b9a-9489-ee3682f37a09-kube-api-access-bclfk\") pod \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.589555 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-model-cache" pod "8673e2ff-31e6-4172-ab13-8e5ea74713cf" (UID: "8673e2ff-31e6-4172-ab13-8e5ea74713cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.590184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.589750 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tmp-dir" pod "40fd49b0-06e5-4b9a-9489-ee3682f37a09" (UID: "40fd49b0-06e5-4b9a-9489-ee3682f37a09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.601294 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.600176 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-home" pod "8673e2ff-31e6-4172-ab13-8e5ea74713cf" (UID: "8673e2ff-31e6-4172-ab13-8e5ea74713cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.601495 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.601468 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tmp-dir" pod "8673e2ff-31e6-4172-ab13-8e5ea74713cf" (UID: "8673e2ff-31e6-4172-ab13-8e5ea74713cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.616313 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.612859 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fd49b0-06e5-4b9a-9489-ee3682f37a09-kube-api-access-bclfk" pod "40fd49b0-06e5-4b9a-9489-ee3682f37a09" (UID: "40fd49b0-06e5-4b9a-9489-ee3682f37a09"). InnerVolumeSpecName "kube-api-access-bclfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:05.616855 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.616829 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tls-certs" pod "8673e2ff-31e6-4172-ab13-8e5ea74713cf" (UID: "8673e2ff-31e6-4172-ab13-8e5ea74713cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:05.616985 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.616939 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8673e2ff-31e6-4172-ab13-8e5ea74713cf-kube-api-access-5cg4b" pod "8673e2ff-31e6-4172-ab13-8e5ea74713cf" (UID: "8673e2ff-31e6-4172-ab13-8e5ea74713cf"). InnerVolumeSpecName "kube-api-access-5cg4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:05.617147 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.617097 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-dshm" pod "8673e2ff-31e6-4172-ab13-8e5ea74713cf" (UID: "8673e2ff-31e6-4172-ab13-8e5ea74713cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.621742 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.621717 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-dshm" pod "40fd49b0-06e5-4b9a-9489-ee3682f37a09" (UID: "40fd49b0-06e5-4b9a-9489-ee3682f37a09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.689575 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.689493 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-home\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-home\") pod \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " Oct 30 19:10:05.689575 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.689537 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-model-cache\") pod \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " Oct 30 19:10:05.689575 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.689562 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tls-certs\" (UniqueName: \"kubernetes.io/secret/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tls-certs\") pod \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\" (UID: \"40fd49b0-06e5-4b9a-9489-ee3682f37a09\") " Oct 30 19:10:05.689892 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.689805 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-model-cache" pod "40fd49b0-06e5-4b9a-9489-ee3682f37a09" (UID: "40fd49b0-06e5-4b9a-9489-ee3682f37a09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.689892 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.689817 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-home" pod "40fd49b0-06e5-4b9a-9489-ee3682f37a09" (UID: "40fd49b0-06e5-4b9a-9489-ee3682f37a09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:05.690014 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.689992 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690076 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690023 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690076 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690037 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690076 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690052 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cg4b\" (UniqueName: \"kubernetes.io/projected/8673e2ff-31e6-4172-ab13-8e5ea74713cf-kube-api-access-5cg4b\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690076 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690069 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690256 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690082 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690256 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690094 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690256 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690106 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690256 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690118 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8673e2ff-31e6-4172-ab13-8e5ea74713cf-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690256 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690131 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bclfk\" (UniqueName: \"kubernetes.io/projected/40fd49b0-06e5-4b9a-9489-ee3682f37a09-kube-api-access-bclfk\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.690256 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.690143 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40fd49b0-06e5-4b9a-9489-ee3682f37a09-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.691884 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.691858 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tls-certs" pod "40fd49b0-06e5-4b9a-9489-ee3682f37a09" (UID: "40fd49b0-06e5-4b9a-9489-ee3682f37a09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:05.706601 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.706569 4862 generic.go:356] "Generic (PLEG): container finished" podID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerID="0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82" exitCode=0 Oct 30 19:10:05.706729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.706675 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" Oct 30 19:10:05.706729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.706704 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" event={"ID":"40fd49b0-06e5-4b9a-9489-ee3682f37a09","Type":"ContainerDied","Data":"0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82"} Oct 30 19:10:05.706808 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.706749 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx" event={"ID":"40fd49b0-06e5-4b9a-9489-ee3682f37a09","Type":"ContainerDied","Data":"7ffa3870e35ccbf36ec4d3c0f302b3a14355a7cf9ee8ec8b6aa0b3d4d3b639ba"} Oct 30 19:10:05.706808 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.706771 4862 scope.go:122] "RemoveContainer" containerID="0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82" Oct 30 19:10:05.709591 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.709567 4862 generic.go:356] "Generic (PLEG): container finished" podID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerID="d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2" exitCode=0 Oct 30 19:10:05.709699 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.709615 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" Oct 30 19:10:05.709699 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.709638 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" event={"ID":"8673e2ff-31e6-4172-ab13-8e5ea74713cf","Type":"ContainerDied","Data":"d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2"} Oct 30 19:10:05.709699 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.709668 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq" event={"ID":"8673e2ff-31e6-4172-ab13-8e5ea74713cf","Type":"ContainerDied","Data":"ef0a385512aa346c6c3cc93fd690af087445993ad50c878abfc96cab5ea50485"} Oct 30 19:10:05.731769 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.731741 4862 scope.go:122] "RemoveContainer" containerID="0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82" Oct 30 19:10:05.735000 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:10:05.732536 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82\": container with ID starting with 0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82 not found: ID does not exist" containerID="0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82" Oct 30 19:10:05.735000 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.732578 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82"} err="failed to get container status \"0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82\": rpc error: code = NotFound desc = could not find container \"0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82\": container with ID starting with 0e503e4eb071751886f81dea2a3962fe1cd0b91a2de2c409a6a06562c319ef82 not found: ID does not exist" Oct 30 19:10:05.735000 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.732604 4862 scope.go:122] "RemoveContainer" containerID="d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2" Oct 30 19:10:05.745419 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.744044 4862 scope.go:122] "RemoveContainer" containerID="d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2" Oct 30 19:10:05.745419 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:10:05.744449 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2\": container with ID starting with d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2 not found: ID does not exist" containerID="d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2" Oct 30 19:10:05.745419 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.744479 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2"} err="failed to get container status \"d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2\": rpc error: code = NotFound desc = could not find container \"d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2\": container with ID starting with d84567ade99a3cb2e2ff11b4efea490ad0828844ae59b770939807a9d08d69e2 not found: ID does not exist" Oct 30 19:10:05.776290 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.775322 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq"] Oct 30 19:10:05.803295 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.795727 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40fd49b0-06e5-4b9a-9489-ee3682f37a09-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:05.803295 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.800936 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/simulator-w1-380c08-kserve-8f555f6bb-qzxhq"] Oct 30 19:10:05.852668 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.835043 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx"] Oct 30 19:10:05.852668 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:05.841355 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-380c08/premium-w1-380c08-kserve-7f9bcfbdc-rn8lx"] Oct 30 19:10:07.063549 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:07.063519 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" path="/var/lib/kubelet/pods/40fd49b0-06e5-4b9a-9489-ee3682f37a09/volumes" Oct 30 19:10:07.063956 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:07.063942 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" path="/var/lib/kubelet/pods/8673e2ff-31e6-4172-ab13-8e5ea74713cf/volumes" Oct 30 19:10:10.931934 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:10.931898 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6"] Oct 30 19:10:10.958898 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:10.958866 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65"] Oct 30 19:10:19.871967 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:10:19.871920 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:10:23.794676 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:23.794645 4862 scope.go:122] "RemoveContainer" containerID="3118f0d85ce6c9e640cb58cb90c3aadf7740b54535fe1ec7e7ba1b818998d6fb" Oct 30 19:10:25.953945 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:25.953904 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" containerID="cri-o://d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0" gracePeriod=45 Oct 30 19:10:25.995258 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:25.995223 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" containerID="cri-o://bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5" gracePeriod=45 Oct 30 19:10:26.283194 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.282850 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 19:10:26.337869 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.337526 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 19:10:26.449465 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449438 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-tmp-dir\") pod \"ae7cb54c-237a-47d1-8b57-151748a41013\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " Oct 30 19:10:26.449670 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449493 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7cb54c-237a-47d1-8b57-151748a41013-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7cb54c-237a-47d1-8b57-151748a41013-tls-certs\") pod \"ae7cb54c-237a-47d1-8b57-151748a41013\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " Oct 30 19:10:26.449670 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449581 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-dshm\") pod \"ae7cb54c-237a-47d1-8b57-151748a41013\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " Oct 30 19:10:26.449670 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449606 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-dshm\") pod \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " Oct 30 19:10:26.449844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449679 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc89dcc-a44d-477a-9570-6f5a7aad5357-kube-api-access-lbnkv\" (UniqueName: \"kubernetes.io/projected/3fc89dcc-a44d-477a-9570-6f5a7aad5357-kube-api-access-lbnkv\") pod \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " Oct 30 19:10:26.449844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449709 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tls-certs\") pod \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " Oct 30 19:10:26.449844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449751 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tmp-dir\") pod \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " Oct 30 19:10:26.449844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449762 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-tmp-dir" pod "ae7cb54c-237a-47d1-8b57-151748a41013" (UID: "ae7cb54c-237a-47d1-8b57-151748a41013"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.449844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449830 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-home\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-home\") pod \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " Oct 30 19:10:26.450383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449903 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7cb54c-237a-47d1-8b57-151748a41013-kube-api-access-m78zr\" (UniqueName: \"kubernetes.io/projected/ae7cb54c-237a-47d1-8b57-151748a41013-kube-api-access-m78zr\") pod \"ae7cb54c-237a-47d1-8b57-151748a41013\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " Oct 30 19:10:26.450383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.449950 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-model-cache\") pod \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\" (UID: \"3fc89dcc-a44d-477a-9570-6f5a7aad5357\") " Oct 30 19:10:26.450383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.450001 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-home\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-home\") pod \"ae7cb54c-237a-47d1-8b57-151748a41013\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " Oct 30 19:10:26.450383 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.450092 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-model-cache\") pod \"ae7cb54c-237a-47d1-8b57-151748a41013\" (UID: \"ae7cb54c-237a-47d1-8b57-151748a41013\") " Oct 30 19:10:26.450822 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.450676 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.451907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.451002 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tmp-dir" pod "3fc89dcc-a44d-477a-9570-6f5a7aad5357" (UID: "3fc89dcc-a44d-477a-9570-6f5a7aad5357"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.451907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.450961 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-model-cache" pod "ae7cb54c-237a-47d1-8b57-151748a41013" (UID: "ae7cb54c-237a-47d1-8b57-151748a41013"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.451907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.451486 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-home" pod "ae7cb54c-237a-47d1-8b57-151748a41013" (UID: "ae7cb54c-237a-47d1-8b57-151748a41013"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.451907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.451871 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-model-cache" pod "3fc89dcc-a44d-477a-9570-6f5a7aad5357" (UID: "3fc89dcc-a44d-477a-9570-6f5a7aad5357"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.452223 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.452058 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-home" pod "3fc89dcc-a44d-477a-9570-6f5a7aad5357" (UID: "3fc89dcc-a44d-477a-9570-6f5a7aad5357"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.452797 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.452773 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tls-certs" pod "3fc89dcc-a44d-477a-9570-6f5a7aad5357" (UID: "3fc89dcc-a44d-477a-9570-6f5a7aad5357"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:26.453043 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.453023 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc89dcc-a44d-477a-9570-6f5a7aad5357-kube-api-access-lbnkv" pod "3fc89dcc-a44d-477a-9570-6f5a7aad5357" (UID: "3fc89dcc-a44d-477a-9570-6f5a7aad5357"). InnerVolumeSpecName "kube-api-access-lbnkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:26.453424 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.453401 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7cb54c-237a-47d1-8b57-151748a41013-tls-certs" pod "ae7cb54c-237a-47d1-8b57-151748a41013" (UID: "ae7cb54c-237a-47d1-8b57-151748a41013"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:26.453535 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.453437 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7cb54c-237a-47d1-8b57-151748a41013-kube-api-access-m78zr" pod "ae7cb54c-237a-47d1-8b57-151748a41013" (UID: "ae7cb54c-237a-47d1-8b57-151748a41013"). InnerVolumeSpecName "kube-api-access-m78zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:26.453686 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.453657 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-dshm" pod "ae7cb54c-237a-47d1-8b57-151748a41013" (UID: "ae7cb54c-237a-47d1-8b57-151748a41013"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.453797 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.453778 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-dshm" pod "3fc89dcc-a44d-477a-9570-6f5a7aad5357" (UID: "3fc89dcc-a44d-477a-9570-6f5a7aad5357"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:26.552781 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552751 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m78zr\" (UniqueName: \"kubernetes.io/projected/ae7cb54c-237a-47d1-8b57-151748a41013-kube-api-access-m78zr\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552785 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552800 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552811 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552823 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7cb54c-237a-47d1-8b57-151748a41013-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552834 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7cb54c-237a-47d1-8b57-151748a41013-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552844 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552855 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnkv\" (UniqueName: \"kubernetes.io/projected/3fc89dcc-a44d-477a-9570-6f5a7aad5357-kube-api-access-lbnkv\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552868 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552927 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.552963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.552938 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc89dcc-a44d-477a-9570-6f5a7aad5357-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:26.853021 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.852930 4862 generic.go:356] "Generic (PLEG): container finished" podID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerID="d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0" exitCode=0 Oct 30 19:10:26.853021 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.852984 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" Oct 30 19:10:26.853313 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.853014 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" event={"ID":"3fc89dcc-a44d-477a-9570-6f5a7aad5357","Type":"ContainerDied","Data":"d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0"} Oct 30 19:10:26.853313 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.853064 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" event={"ID":"3fc89dcc-a44d-477a-9570-6f5a7aad5357","Type":"ContainerDied","Data":"9296e502b28f9848bec43fb3096cd75d6c3a66ca4b9ece61c8eb262f12bbf7b1"} Oct 30 19:10:26.853313 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.853082 4862 scope.go:122] "RemoveContainer" containerID="d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0" Oct 30 19:10:26.854470 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.854449 4862 generic.go:356] "Generic (PLEG): container finished" podID="ae7cb54c-237a-47d1-8b57-151748a41013" containerID="bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5" exitCode=0 Oct 30 19:10:26.854580 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.854498 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" Oct 30 19:10:26.854580 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.854538 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" event={"ID":"ae7cb54c-237a-47d1-8b57-151748a41013","Type":"ContainerDied","Data":"bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5"} Oct 30 19:10:26.854682 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.854576 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65" event={"ID":"ae7cb54c-237a-47d1-8b57-151748a41013","Type":"ContainerDied","Data":"c638c1cc07d81b3b6e507456cdc80204457e44a5a94a1c4498c62fb5393e314d"} Oct 30 19:10:26.864706 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.864688 4862 scope.go:122] "RemoveContainer" containerID="d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0" Oct 30 19:10:26.865006 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:10:26.864983 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0\": container with ID starting with d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0 not found: ID does not exist" containerID="d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0" Oct 30 19:10:26.865094 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.865017 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0"} err="failed to get container status \"d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0\": rpc error: code = NotFound desc = could not find container \"d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0\": container with ID starting with d393ae75f5e967ec360ec79a2ebd4f747be5b98db056258e8c90f94e63157cb0 not found: ID does not exist" Oct 30 19:10:26.865094 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.865040 4862 scope.go:122] "RemoveContainer" containerID="bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5" Oct 30 19:10:26.876149 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.876126 4862 scope.go:122] "RemoveContainer" containerID="bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5" Oct 30 19:10:26.876758 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:10:26.876734 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5\": container with ID starting with bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5 not found: ID does not exist" containerID="bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5" Oct 30 19:10:26.876841 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.876765 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5"} err="failed to get container status \"bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5\": rpc error: code = NotFound desc = could not find container \"bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5\": container with ID starting with bb231cb784d629a826734cc1f57a157cf0cdda97d2c50f13b433c0bd4d8c30a5 not found: ID does not exist" Oct 30 19:10:26.899417 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.899391 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65"] Oct 30 19:10:26.909660 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.909638 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/simulator-w3-de7e3a-kserve-686c95ff4c-h9b65"] Oct 30 19:10:26.932384 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.932352 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6"] Oct 30 19:10:26.936243 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:26.936219 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6"] Oct 30 19:10:27.063891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:27.063857 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" path="/var/lib/kubelet/pods/3fc89dcc-a44d-477a-9570-6f5a7aad5357/volumes" Oct 30 19:10:27.064350 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:27.064334 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" path="/var/lib/kubelet/pods/ae7cb54c-237a-47d1-8b57-151748a41013/volumes" Oct 30 19:10:27.191641 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:27.191503 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-de7e3a/distinct-w3-de7e3a-kserve-9b8594ddf-2mcz6" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:10:30.060783 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.060755 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744"] Oct 30 19:10:30.061373 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061354 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" Oct 30 19:10:30.061429 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061376 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" Oct 30 19:10:30.061429 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061401 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" Oct 30 19:10:30.061429 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061409 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" Oct 30 19:10:30.061429 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061424 4862 cpu_manager.go:414] "containerMap: removing container" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" Oct 30 19:10:30.061567 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061432 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" Oct 30 19:10:30.061567 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061474 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" Oct 30 19:10:30.061567 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061480 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" Oct 30 19:10:30.061567 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061492 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5cc43bf9-1f43-473c-be5b-7130e798afd3" containerName="revoke-keys" Oct 30 19:10:30.061567 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061499 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc43bf9-1f43-473c-be5b-7130e798afd3" containerName="revoke-keys" Oct 30 19:10:30.061743 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061617 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc89dcc-a44d-477a-9570-6f5a7aad5357" containerName="main" Oct 30 19:10:30.061743 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061627 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc43bf9-1f43-473c-be5b-7130e798afd3" containerName="revoke-keys" Oct 30 19:10:30.061743 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061635 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8673e2ff-31e6-4172-ab13-8e5ea74713cf" containerName="main" Oct 30 19:10:30.061743 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061642 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7cb54c-237a-47d1-8b57-151748a41013" containerName="main" Oct 30 19:10:30.061743 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.061649 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="40fd49b0-06e5-4b9a-9489-ee3682f37a09" containerName="main" Oct 30 19:10:30.064904 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.064878 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.094192 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.094165 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744"] Oct 30 19:10:30.124556 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.124527 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.124726 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.124636 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-certs\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.124726 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.124720 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320d4612-0fc7-411d-8fd7-a1874bf84231-istiod-ca-cert\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.124867 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.124749 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ltj\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-kube-api-access-s2ltj\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.124867 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.124849 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-envoy\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.125054 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.125029 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-credential-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.125143 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.125070 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-data\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.125221 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.125167 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-token\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.125221 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.125201 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-podinfo\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.158615 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.158595 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-09ef1d-openshift-default-dockercfg-rml7c\"" Oct 30 19:10:30.226577 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.226554 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-credential-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.226715 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.226593 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-data\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.226715 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.226640 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-token\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.226715 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.226664 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-podinfo\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.226908 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.226889 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.226992 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.226931 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-credential-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.226992 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.226978 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-certs\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.227069 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.227046 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320d4612-0fc7-411d-8fd7-a1874bf84231-istiod-ca-cert\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.227112 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.227077 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ltj\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-kube-api-access-s2ltj\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.227112 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.227098 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-envoy\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.227234 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.227045 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-data\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.227335 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.227235 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-socket\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.227335 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.227314 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-certs\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.227662 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.227642 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320d4612-0fc7-411d-8fd7-a1874bf84231-istiod-ca-cert\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.229320 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.229301 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-envoy\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.229449 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.229430 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-podinfo\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.247758 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.247738 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-token\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.247849 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.247789 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ltj\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-kube-api-access-s2ltj\") pod \"e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.380564 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.380511 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:30.698547 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.698516 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744"] Oct 30 19:10:30.700169 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:10:30.700140 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320d4612_0fc7_411d_8fd7_a1874bf84231.slice/crio-9fffcd56a5c2aa663521c61a853865d946e6c6e33fd105e657cc186e1eec7221 WatchSource:0}: Error finding container 9fffcd56a5c2aa663521c61a853865d946e6c6e33fd105e657cc186e1eec7221: Status 404 returned error can't find the container with id 9fffcd56a5c2aa663521c61a853865d946e6c6e33fd105e657cc186e1eec7221 Oct 30 19:10:30.702902 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.702872 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:10:30.702980 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.702933 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:10:30.702980 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.702968 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:10:30.910336 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.910296 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" event={"ID":"320d4612-0fc7-411d-8fd7-a1874bf84231","Type":"ContainerStarted","Data":"e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd"} Oct 30 19:10:30.910517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.910358 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" event={"ID":"320d4612-0fc7-411d-8fd7-a1874bf84231","Type":"ContainerStarted","Data":"9fffcd56a5c2aa663521c61a853865d946e6c6e33fd105e657cc186e1eec7221"} Oct 30 19:10:30.950434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:30.950336 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podStartSLOduration=0.950322846 podStartE2EDuration="950.322846ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:30.94938283 +0000 UTC m=+4329.998427743" watchObservedRunningTime="2026-10-30 19:10:30.950322846 +0000 UTC m=+4329.999367759" Oct 30 19:10:31.381521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:31.381486 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:31.382857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:31.382833 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 30 19:10:31.382977 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:31.382878 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 30 19:10:32.381452 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:32.381417 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 30 19:10:32.381634 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:32.381468 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 30 19:10:33.381017 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:33.380984 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 30 19:10:33.381255 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:33.381047 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 30 19:10:34.381535 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:34.381501 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 30 19:10:34.381930 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:34.381551 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 30 19:10:35.380983 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:35.380950 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 30 19:10:35.381151 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:35.381004 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 30 19:10:35.654221 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:35.654133 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l"] Oct 30 19:10:35.655131 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:35.655054 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" podUID="83501714-680a-4c27-bfef-fd5e65bc12c9" containerName="maas-api" containerID="cri-o://0444f4eff4eb32ac10fd2717a48e92ec35b35fe22c86eaa0bedbce3c90cb8331" gracePeriod=30 Oct 30 19:10:36.088486 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.088448 4862 generic.go:356] "Generic (PLEG): container finished" podID="83501714-680a-4c27-bfef-fd5e65bc12c9" containerID="0444f4eff4eb32ac10fd2717a48e92ec35b35fe22c86eaa0bedbce3c90cb8331" exitCode=0 Oct 30 19:10:36.088940 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.088574 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" event={"ID":"83501714-680a-4c27-bfef-fd5e65bc12c9","Type":"ContainerDied","Data":"0444f4eff4eb32ac10fd2717a48e92ec35b35fe22c86eaa0bedbce3c90cb8331"} Oct 30 19:10:36.116826 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.116795 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h"] Oct 30 19:10:36.121966 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.121947 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.139521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.139491 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 19:10:36.139882 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.139859 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:10:36.140609 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.140543 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:10:36.209300 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.205973 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h"] Oct 30 19:10:36.268571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.254142 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83501714-680a-4c27-bfef-fd5e65bc12c9-kube-api-access-5b4gc\" (UniqueName: \"kubernetes.io/projected/83501714-680a-4c27-bfef-fd5e65bc12c9-kube-api-access-5b4gc\") pod \"83501714-680a-4c27-bfef-fd5e65bc12c9\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " Oct 30 19:10:36.268571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.254205 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-metrics-certs\") pod \"83501714-680a-4c27-bfef-fd5e65bc12c9\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " Oct 30 19:10:36.268571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.254277 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83501714-680a-4c27-bfef-fd5e65bc12c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83501714-680a-4c27-bfef-fd5e65bc12c9-openshift-service-ca\") pod \"83501714-680a-4c27-bfef-fd5e65bc12c9\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " Oct 30 19:10:36.268571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.254337 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-maas-api-tls\") pod \"83501714-680a-4c27-bfef-fd5e65bc12c9\" (UID: \"83501714-680a-4c27-bfef-fd5e65bc12c9\") " Oct 30 19:10:36.268571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.254879 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70169aa9-g9k2h\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.268571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.254938 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvs5\" (UniqueName: \"kubernetes.io/projected/e4c9d7f7-d869-4565-88d5-44610a2c8b55-kube-api-access-mlvs5\") pod \"maas-api-revoke-sub-70169aa9-g9k2h\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.268571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.255030 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70169aa9-g9k2h\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.314425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.310624 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83501714-680a-4c27-bfef-fd5e65bc12c9-openshift-service-ca" pod "83501714-680a-4c27-bfef-fd5e65bc12c9" (UID: "83501714-680a-4c27-bfef-fd5e65bc12c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:36.315604 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.315550 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83501714-680a-4c27-bfef-fd5e65bc12c9-kube-api-access-5b4gc" pod "83501714-680a-4c27-bfef-fd5e65bc12c9" (UID: "83501714-680a-4c27-bfef-fd5e65bc12c9"). InnerVolumeSpecName "kube-api-access-5b4gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:36.321775 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.321747 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-maas-api-tls" pod "83501714-680a-4c27-bfef-fd5e65bc12c9" (UID: "83501714-680a-4c27-bfef-fd5e65bc12c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:36.322000 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.321950 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-metrics-certs" pod "83501714-680a-4c27-bfef-fd5e65bc12c9" (UID: "83501714-680a-4c27-bfef-fd5e65bc12c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:36.378909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.376938 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70169aa9-g9k2h\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.378909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.376993 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvs5\" (UniqueName: \"kubernetes.io/projected/e4c9d7f7-d869-4565-88d5-44610a2c8b55-kube-api-access-mlvs5\") pod \"maas-api-revoke-sub-70169aa9-g9k2h\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.378909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.377052 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70169aa9-g9k2h\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.378909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.377166 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4gc\" (UniqueName: \"kubernetes.io/projected/83501714-680a-4c27-bfef-fd5e65bc12c9-kube-api-access-5b4gc\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:36.378909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.377177 4862 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-metrics-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:36.378909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.377186 4862 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83501714-680a-4c27-bfef-fd5e65bc12c9-openshift-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:36.378909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.377195 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83501714-680a-4c27-bfef-fd5e65bc12c9-maas-api-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:36.378909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.378210 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70169aa9-g9k2h\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.380565 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.380516 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70169aa9-g9k2h\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.384412 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.384383 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 30 19:10:36.384563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.384450 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 30 19:10:36.396403 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.396375 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvs5\" (UniqueName: \"kubernetes.io/projected/e4c9d7f7-d869-4565-88d5-44610a2c8b55-kube-api-access-mlvs5\") pod \"maas-api-revoke-sub-70169aa9-g9k2h\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.466448 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.466412 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:36.561371 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.561336 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk"] Oct 30 19:10:36.586080 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.586040 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-380c08-n9dvk"] Oct 30 19:10:36.785299 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:36.785246 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h"] Oct 30 19:10:37.064688 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:37.064653 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d775d3-579f-4166-8929-165b759b8df2" path="/var/lib/kubelet/pods/c4d775d3-579f-4166-8929-165b759b8df2/volumes" Oct 30 19:10:37.097165 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:37.097141 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" Oct 30 19:10:37.097331 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:37.097180 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l" event={"ID":"83501714-680a-4c27-bfef-fd5e65bc12c9","Type":"ContainerDied","Data":"31df38f208df065536d58a3f7b03ccb9810dcb54f7baf33e4408d2ea9e2ae800"} Oct 30 19:10:37.097331 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:37.097253 4862 scope.go:122] "RemoveContainer" containerID="0444f4eff4eb32ac10fd2717a48e92ec35b35fe22c86eaa0bedbce3c90cb8331" Oct 30 19:10:37.098841 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:37.098806 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" event={"ID":"e4c9d7f7-d869-4565-88d5-44610a2c8b55","Type":"ContainerStarted","Data":"25ec0acf9cf80d2b761839420628aef20949b3af5e7e5d7a2e39cb47a237f10a"} Oct 30 19:10:37.159192 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:37.159149 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l"] Oct 30 19:10:37.161901 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:37.161881 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-380c08-765447b4f-9sg6l"] Oct 30 19:10:37.382193 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:37.382114 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body= Oct 30 19:10:37.382193 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:37.382167 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" Oct 30 19:10:38.116949 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:38.116922 4862 generic.go:356] "Generic (PLEG): container finished" podID="e4c9d7f7-d869-4565-88d5-44610a2c8b55" containerID="45c7dbeb6b92e52d48a396311013ff70c89e3014c478a4c764cf0b8e915bfbb6" exitCode=0 Oct 30 19:10:38.117356 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:38.117012 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" event={"ID":"e4c9d7f7-d869-4565-88d5-44610a2c8b55","Type":"ContainerDied","Data":"45c7dbeb6b92e52d48a396311013ff70c89e3014c478a4c764cf0b8e915bfbb6"} Oct 30 19:10:38.385285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:38.385232 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:38.385699 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:38.385676 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:38.386361 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:38.386344 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:10:38.576986 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:38.576948 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:10:39.063633 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.063597 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83501714-680a-4c27-bfef-fd5e65bc12c9" path="/var/lib/kubelet/pods/83501714-680a-4c27-bfef-fd5e65bc12c9/volumes" Oct 30 19:10:39.398816 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.398784 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:39.530135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.528691 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-openshift-ca\") pod \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " Oct 30 19:10:39.530373 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.530214 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-xks-ca\") pod \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " Oct 30 19:10:39.530373 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.530312 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c9d7f7-d869-4565-88d5-44610a2c8b55-kube-api-access-mlvs5\" (UniqueName: \"kubernetes.io/projected/e4c9d7f7-d869-4565-88d5-44610a2c8b55-kube-api-access-mlvs5\") pod \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\" (UID: \"e4c9d7f7-d869-4565-88d5-44610a2c8b55\") " Oct 30 19:10:39.532090 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.531886 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-openshift-ca" pod "e4c9d7f7-d869-4565-88d5-44610a2c8b55" (UID: "e4c9d7f7-d869-4565-88d5-44610a2c8b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:39.536872 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.536818 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c9d7f7-d869-4565-88d5-44610a2c8b55-kube-api-access-mlvs5" pod "e4c9d7f7-d869-4565-88d5-44610a2c8b55" (UID: "e4c9d7f7-d869-4565-88d5-44610a2c8b55"). InnerVolumeSpecName "kube-api-access-mlvs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:39.537022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.536995 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-xks-ca" pod "e4c9d7f7-d869-4565-88d5-44610a2c8b55" (UID: "e4c9d7f7-d869-4565-88d5-44610a2c8b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:10:39.632653 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.632582 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvs5\" (UniqueName: \"kubernetes.io/projected/e4c9d7f7-d869-4565-88d5-44610a2c8b55-kube-api-access-mlvs5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:39.632653 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.632610 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:39.632653 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:39.632620 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c9d7f7-d869-4565-88d5-44610a2c8b55-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:40.135160 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:40.135129 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" Oct 30 19:10:40.135384 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:40.135165 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h" event={"ID":"e4c9d7f7-d869-4565-88d5-44610a2c8b55","Type":"ContainerDied","Data":"25ec0acf9cf80d2b761839420628aef20949b3af5e7e5d7a2e39cb47a237f10a"} Oct 30 19:10:40.135384 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:40.135225 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25ec0acf9cf80d2b761839420628aef20949b3af5e7e5d7a2e39cb47a237f10a" Oct 30 19:10:44.403661 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:44.398089 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc"] Oct 30 19:10:44.403661 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:44.398364 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" containerID="cri-o://8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b" gracePeriod=30 Oct 30 19:10:46.244810 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:46.244775 4862 patch_prober.go:29] interesting pod/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:10:46.245215 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:46.244834 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:10:49.680633 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.680567 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 19:10:49.776481 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776451 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-kube-api-access-4jrvz\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-kube-api-access-4jrvz\") pod \"5a3b36b7-bcaa-40be-a051-8694cd685308\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " Oct 30 19:10:49.776667 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776518 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-token\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-token\") pod \"5a3b36b7-bcaa-40be-a051-8694cd685308\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " Oct 30 19:10:49.776667 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776542 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-socket\") pod \"5a3b36b7-bcaa-40be-a051-8694cd685308\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " Oct 30 19:10:49.776667 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776577 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3b36b7-bcaa-40be-a051-8694cd685308-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3b36b7-bcaa-40be-a051-8694cd685308-istiod-ca-cert\") pod \"5a3b36b7-bcaa-40be-a051-8694cd685308\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " Oct 30 19:10:49.776667 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776661 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-podinfo\") pod \"5a3b36b7-bcaa-40be-a051-8694cd685308\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " Oct 30 19:10:49.776915 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776689 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-credential-socket\") pod \"5a3b36b7-bcaa-40be-a051-8694cd685308\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " Oct 30 19:10:49.776915 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776821 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-certs\") pod \"5a3b36b7-bcaa-40be-a051-8694cd685308\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " Oct 30 19:10:49.776915 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776859 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-envoy\") pod \"5a3b36b7-bcaa-40be-a051-8694cd685308\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " Oct 30 19:10:49.776915 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776908 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-data\") pod \"5a3b36b7-bcaa-40be-a051-8694cd685308\" (UID: \"5a3b36b7-bcaa-40be-a051-8694cd685308\") " Oct 30 19:10:49.777149 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.776897 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-socket" pod "5a3b36b7-bcaa-40be-a051-8694cd685308" (UID: "5a3b36b7-bcaa-40be-a051-8694cd685308"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:49.777149 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.777138 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-certs" pod "5a3b36b7-bcaa-40be-a051-8694cd685308" (UID: "5a3b36b7-bcaa-40be-a051-8694cd685308"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:49.777414 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.777349 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-data" pod "5a3b36b7-bcaa-40be-a051-8694cd685308" (UID: "5a3b36b7-bcaa-40be-a051-8694cd685308"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:49.777414 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.777371 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-credential-socket" pod "5a3b36b7-bcaa-40be-a051-8694cd685308" (UID: "5a3b36b7-bcaa-40be-a051-8694cd685308"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:49.777674 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.777636 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:49.777674 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.777657 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:49.777674 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.777672 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:49.777850 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.777685 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:49.777976 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.777951 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3b36b7-bcaa-40be-a051-8694cd685308-istiod-ca-cert" pod "5a3b36b7-bcaa-40be-a051-8694cd685308" (UID: "5a3b36b7-bcaa-40be-a051-8694cd685308"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:49.779213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.779192 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-token" pod "5a3b36b7-bcaa-40be-a051-8694cd685308" (UID: "5a3b36b7-bcaa-40be-a051-8694cd685308"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:49.779829 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.779808 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-podinfo" pod "5a3b36b7-bcaa-40be-a051-8694cd685308" (UID: "5a3b36b7-bcaa-40be-a051-8694cd685308"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:10:49.779912 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.779878 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-kube-api-access-4jrvz" pod "5a3b36b7-bcaa-40be-a051-8694cd685308" (UID: "5a3b36b7-bcaa-40be-a051-8694cd685308"). InnerVolumeSpecName "kube-api-access-4jrvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:49.780371 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.780341 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-envoy" pod "5a3b36b7-bcaa-40be-a051-8694cd685308" (UID: "5a3b36b7-bcaa-40be-a051-8694cd685308"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:10:49.871573 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:10:49.871537 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:10:49.879022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.878994 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:49.879022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.879022 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:49.879209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.879036 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrvz\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-kube-api-access-4jrvz\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:49.879209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.879048 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3b36b7-bcaa-40be-a051-8694cd685308-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:49.879209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:49.879060 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3b36b7-bcaa-40be-a051-8694cd685308-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:10:50.221847 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:50.221814 4862 generic.go:356] "Generic (PLEG): container finished" podID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerID="8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b" exitCode=0 Oct 30 19:10:50.222059 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:50.221862 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" Oct 30 19:10:50.222059 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:50.221902 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" event={"ID":"5a3b36b7-bcaa-40be-a051-8694cd685308","Type":"ContainerDied","Data":"8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b"} Oct 30 19:10:50.222059 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:50.221966 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc" event={"ID":"5a3b36b7-bcaa-40be-a051-8694cd685308","Type":"ContainerDied","Data":"584adebf2a13a2d2330bd1f10be3a05216611fff9dc25a60d6704079bd2b4eb5"} Oct 30 19:10:50.222059 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:50.221995 4862 scope.go:122] "RemoveContainer" containerID="8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b" Oct 30 19:10:50.236642 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:50.236625 4862 scope.go:122] "RemoveContainer" containerID="8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b" Oct 30 19:10:50.236933 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:10:50.236910 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b\": container with ID starting with 8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b not found: ID does not exist" containerID="8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b" Oct 30 19:10:50.237012 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:50.236944 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b"} err="failed to get container status \"8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b\": rpc error: code = NotFound desc = could not find container \"8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b\": container with ID starting with 8ce7fcf049d973613a09751cc048196e0f805963a4bd23e9d709015f2524179b not found: ID does not exist" Oct 30 19:10:50.270105 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:50.270081 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc"] Oct 30 19:10:50.280108 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:50.280087 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-380c08-openshift-default-598cf58675-wrmvc"] Oct 30 19:10:51.062143 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:10:51.062112 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" path="/var/lib/kubelet/pods/5a3b36b7-bcaa-40be-a051-8694cd685308/volumes" Oct 30 19:11:15.008121 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.008040 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc"] Oct 30 19:11:15.008692 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.008326 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" podUID="074da152-8cdb-46c0-bf73-d800dd4355ff" containerName="discovery" containerID="cri-o://72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f" gracePeriod=30 Oct 30 19:11:15.278960 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.278892 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 19:11:15.408293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.408230 4862 generic.go:356] "Generic (PLEG): container finished" podID="074da152-8cdb-46c0-bf73-d800dd4355ff" containerID="72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f" exitCode=0 Oct 30 19:11:15.408467 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.408299 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" Oct 30 19:11:15.408467 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.408319 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" event={"ID":"074da152-8cdb-46c0-bf73-d800dd4355ff","Type":"ContainerDied","Data":"72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f"} Oct 30 19:11:15.408467 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.408378 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc" event={"ID":"074da152-8cdb-46c0-bf73-d800dd4355ff","Type":"ContainerDied","Data":"6b5bf6be609269a447e123e54e5eea230fdaed3c9ab92ca493fcdf92406b9c40"} Oct 30 19:11:15.408467 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.408405 4862 scope.go:122] "RemoveContainer" containerID="72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f" Oct 30 19:11:15.412894 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.412872 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-ca-configmap\") pod \"074da152-8cdb-46c0-bf73-d800dd4355ff\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " Oct 30 19:11:15.413012 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.412940 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-cacerts\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-cacerts\") pod \"074da152-8cdb-46c0-bf73-d800dd4355ff\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " Oct 30 19:11:15.413096 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.413080 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-kube-api-access-h6g5q\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-kube-api-access-h6g5q\") pod \"074da152-8cdb-46c0-bf73-d800dd4355ff\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " Oct 30 19:11:15.413189 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.413143 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-dns-cert\") pod \"074da152-8cdb-46c0-bf73-d800dd4355ff\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " Oct 30 19:11:15.413253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.413192 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-token\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-token\") pod \"074da152-8cdb-46c0-bf73-d800dd4355ff\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " Oct 30 19:11:15.413253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.413221 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-kubeconfig\") pod \"074da152-8cdb-46c0-bf73-d800dd4355ff\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " Oct 30 19:11:15.413391 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.413321 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074da152-8cdb-46c0-bf73-d800dd4355ff-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/074da152-8cdb-46c0-bf73-d800dd4355ff-local-certs\") pod \"074da152-8cdb-46c0-bf73-d800dd4355ff\" (UID: \"074da152-8cdb-46c0-bf73-d800dd4355ff\") " Oct 30 19:11:15.413391 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.413359 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-ca-configmap" pod "074da152-8cdb-46c0-bf73-d800dd4355ff" (UID: "074da152-8cdb-46c0-bf73-d800dd4355ff"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:15.414173 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.414143 4862 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-ca-configmap\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:15.415589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.415553 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-kube-api-access-h6g5q" pod "074da152-8cdb-46c0-bf73-d800dd4355ff" (UID: "074da152-8cdb-46c0-bf73-d800dd4355ff"). InnerVolumeSpecName "kube-api-access-h6g5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:15.416060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.416039 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074da152-8cdb-46c0-bf73-d800dd4355ff-local-certs" pod "074da152-8cdb-46c0-bf73-d800dd4355ff" (UID: "074da152-8cdb-46c0-bf73-d800dd4355ff"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:11:15.416226 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.416204 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-dns-cert" pod "074da152-8cdb-46c0-bf73-d800dd4355ff" (UID: "074da152-8cdb-46c0-bf73-d800dd4355ff"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:15.416350 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.416250 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-token" pod "074da152-8cdb-46c0-bf73-d800dd4355ff" (UID: "074da152-8cdb-46c0-bf73-d800dd4355ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:15.417076 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.417054 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-cacerts" pod "074da152-8cdb-46c0-bf73-d800dd4355ff" (UID: "074da152-8cdb-46c0-bf73-d800dd4355ff"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:15.417627 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.417603 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-kubeconfig" pod "074da152-8cdb-46c0-bf73-d800dd4355ff" (UID: "074da152-8cdb-46c0-bf73-d800dd4355ff"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:15.446098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.445749 4862 scope.go:122] "RemoveContainer" containerID="72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f" Oct 30 19:11:15.446098 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:11:15.446080 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f\": container with ID starting with 72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f not found: ID does not exist" containerID="72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f" Oct 30 19:11:15.446286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.446114 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f"} err="failed to get container status \"72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f\": rpc error: code = NotFound desc = could not find container \"72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f\": container with ID starting with 72a6a1fc8d153ecdc6b2eb67db4ab080c9a5a610c23d4290ea2bd6c2827a682f not found: ID does not exist" Oct 30 19:11:15.515376 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.515346 4862 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-csr-dns-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:15.515376 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.515372 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:15.515552 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.515384 4862 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-istio-kubeconfig\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:15.515552 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.515392 4862 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/074da152-8cdb-46c0-bf73-d800dd4355ff-local-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:15.515552 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.515401 4862 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/074da152-8cdb-46c0-bf73-d800dd4355ff-cacerts\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:15.515552 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.515408 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6g5q\" (UniqueName: \"kubernetes.io/projected/074da152-8cdb-46c0-bf73-d800dd4355ff-kube-api-access-h6g5q\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:15.751255 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.751230 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc"] Oct 30 19:11:15.761154 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:15.761126 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-d74mc"] Oct 30 19:11:17.061829 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:17.061793 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074da152-8cdb-46c0-bf73-d800dd4355ff" path="/var/lib/kubelet/pods/074da152-8cdb-46c0-bf73-d800dd4355ff/volumes" Oct 30 19:11:19.877200 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:11:19.877162 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:11:23.037206 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:23.037168 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:11:23.842064 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:23.842018 4862 scope.go:122] "RemoveContainer" containerID="73146b833099fc6054e74b19293aa2a0cb5bbaf15716cbee5ca9268a59191f30" Oct 30 19:11:23.854538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:23.854512 4862 scope.go:122] "RemoveContainer" containerID="0e751a5d286604bcaa90565c8a48097ec8d676d61ccbe68e1c6ac7dec8bbf1d1" Oct 30 19:11:23.865470 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:23.865445 4862 scope.go:122] "RemoveContainer" containerID="7e379d82a602859e2776f7a3ab955017a03e8ccd7883b7d5e833527f960e120c" Oct 30 19:11:23.881201 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:23.881178 4862 scope.go:122] "RemoveContainer" containerID="76add42882a48dec73facc3e7aff5bd347f7e0329e0a2a4bfa3149015f852782" Oct 30 19:11:23.891100 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:23.891070 4862 scope.go:122] "RemoveContainer" containerID="57aca73380c7814b3901a3198af2ae38baffbe9519627ddaaa9c3674da28623d" Oct 30 19:11:23.899679 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:23.899660 4862 scope.go:122] "RemoveContainer" containerID="f9a3b51dc019e4e899f3dd9eaaade35fc77822147a0783a0d6b0633e3addffb8" Oct 30 19:11:36.027789 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:36.027752 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z"] Oct 30 19:11:36.037347 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:36.037319 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0356d9d-v492z"] Oct 30 19:11:37.064609 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:37.064567 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefea2ca-0c8a-4ade-93f4-35042318fa38" path="/var/lib/kubelet/pods/cefea2ca-0c8a-4ade-93f4-35042318fa38/volumes" Oct 30 19:11:48.766651 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:48.766619 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:11:49.882095 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:11:49.882057 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:11:52.713251 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713219 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg"] Oct 30 19:11:52.713737 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713723 4862 cpu_manager.go:414] "containerMap: removing container" podUID="83501714-680a-4c27-bfef-fd5e65bc12c9" containerName="maas-api" Oct 30 19:11:52.713787 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713739 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83501714-680a-4c27-bfef-fd5e65bc12c9" containerName="maas-api" Oct 30 19:11:52.713787 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713754 4862 cpu_manager.go:414] "containerMap: removing container" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" Oct 30 19:11:52.713787 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713759 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" Oct 30 19:11:52.713787 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713774 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e4c9d7f7-d869-4565-88d5-44610a2c8b55" containerName="revoke-keys" Oct 30 19:11:52.713787 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713779 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c9d7f7-d869-4565-88d5-44610a2c8b55" containerName="revoke-keys" Oct 30 19:11:52.713991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713800 4862 cpu_manager.go:414] "containerMap: removing container" podUID="074da152-8cdb-46c0-bf73-d800dd4355ff" containerName="discovery" Oct 30 19:11:52.713991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713805 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074da152-8cdb-46c0-bf73-d800dd4355ff" containerName="discovery" Oct 30 19:11:52.713991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713879 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="074da152-8cdb-46c0-bf73-d800dd4355ff" containerName="discovery" Oct 30 19:11:52.713991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713890 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="83501714-680a-4c27-bfef-fd5e65bc12c9" containerName="maas-api" Oct 30 19:11:52.713991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713897 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3b36b7-bcaa-40be-a051-8694cd685308" containerName="istio-proxy" Oct 30 19:11:52.713991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.713905 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c9d7f7-d869-4565-88d5-44610a2c8b55" containerName="revoke-keys" Oct 30 19:11:52.718414 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.718396 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:52.742449 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.742426 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:11:52.742691 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.742665 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:11:52.745533 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.745507 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg"] Oct 30 19:11:52.851593 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.851565 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv8bz\" (UniqueName: \"kubernetes.io/projected/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-kube-api-access-pv8bz\") pod \"maas-api-revoke-sub-8b0ee064-scspg\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:52.851749 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.851654 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b0ee064-scspg\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:52.851792 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.851785 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b0ee064-scspg\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:52.952780 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.952753 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b0ee064-scspg\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:52.952933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.952832 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b0ee064-scspg\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:52.952933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.952898 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv8bz\" (UniqueName: \"kubernetes.io/projected/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-kube-api-access-pv8bz\") pod \"maas-api-revoke-sub-8b0ee064-scspg\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:52.953576 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.953552 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b0ee064-scspg\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:52.955244 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.955223 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b0ee064-scspg\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:52.970851 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:52.970799 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv8bz\" (UniqueName: \"kubernetes.io/projected/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-kube-api-access-pv8bz\") pod \"maas-api-revoke-sub-8b0ee064-scspg\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:53.031353 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:53.031330 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:53.333868 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:53.333845 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg"] Oct 30 19:11:53.336378 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:11:53.336342 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eaffcd7_2ea0_4a5c_ad87_90a65c976251.slice/crio-c1652e2b36293b8c470dd47f23c0f593b59aa4a92aabeee7ec6c73333a929e35 WatchSource:0}: Error finding container c1652e2b36293b8c470dd47f23c0f593b59aa4a92aabeee7ec6c73333a929e35: Status 404 returned error can't find the container with id c1652e2b36293b8c470dd47f23c0f593b59aa4a92aabeee7ec6c73333a929e35 Oct 30 19:11:53.682817 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:53.682772 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" event={"ID":"0eaffcd7-2ea0-4a5c-ad87-90a65c976251","Type":"ContainerStarted","Data":"cd296aed642f2201826d33855866af38aa750b6b1c7d97cde77a30bbb66f875d"} Oct 30 19:11:53.682959 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:53.682842 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" event={"ID":"0eaffcd7-2ea0-4a5c-ad87-90a65c976251","Type":"ContainerStarted","Data":"c1652e2b36293b8c470dd47f23c0f593b59aa4a92aabeee7ec6c73333a929e35"} Oct 30 19:11:53.720535 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:53.720490 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" podStartSLOduration=1.72047517 podStartE2EDuration="1.72047517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:11:53.717479967 +0000 UTC m=+4412.766524880" watchObservedRunningTime="2026-10-30 19:11:53.72047517 +0000 UTC m=+4412.769520079" Oct 30 19:11:54.690930 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:54.690891 4862 generic.go:356] "Generic (PLEG): container finished" podID="0eaffcd7-2ea0-4a5c-ad87-90a65c976251" containerID="cd296aed642f2201826d33855866af38aa750b6b1c7d97cde77a30bbb66f875d" exitCode=0 Oct 30 19:11:54.691116 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:54.690970 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" event={"ID":"0eaffcd7-2ea0-4a5c-ad87-90a65c976251","Type":"ContainerDied","Data":"cd296aed642f2201826d33855866af38aa750b6b1c7d97cde77a30bbb66f875d"} Oct 30 19:11:55.849710 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:55.849684 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:55.988738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:55.988668 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-xks-ca\") pod \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " Oct 30 19:11:55.988738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:55.988712 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-openshift-ca\") pod \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " Oct 30 19:11:55.988948 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:55.988755 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-kube-api-access-pv8bz\" (UniqueName: \"kubernetes.io/projected/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-kube-api-access-pv8bz\") pod \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\" (UID: \"0eaffcd7-2ea0-4a5c-ad87-90a65c976251\") " Oct 30 19:11:55.989156 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:55.989133 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-openshift-ca" pod "0eaffcd7-2ea0-4a5c-ad87-90a65c976251" (UID: "0eaffcd7-2ea0-4a5c-ad87-90a65c976251"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:11:55.989301 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:55.989210 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:55.991069 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:55.991039 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-xks-ca" pod "0eaffcd7-2ea0-4a5c-ad87-90a65c976251" (UID: "0eaffcd7-2ea0-4a5c-ad87-90a65c976251"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:11:55.991185 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:55.991167 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-kube-api-access-pv8bz" pod "0eaffcd7-2ea0-4a5c-ad87-90a65c976251" (UID: "0eaffcd7-2ea0-4a5c-ad87-90a65c976251"). InnerVolumeSpecName "kube-api-access-pv8bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:11:56.090197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:56.090174 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv8bz\" (UniqueName: \"kubernetes.io/projected/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-kube-api-access-pv8bz\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:56.090197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:56.090197 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eaffcd7-2ea0-4a5c-ad87-90a65c976251-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:11:56.705319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:56.705286 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" Oct 30 19:11:56.705319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:56.705310 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg" event={"ID":"0eaffcd7-2ea0-4a5c-ad87-90a65c976251","Type":"ContainerDied","Data":"c1652e2b36293b8c470dd47f23c0f593b59aa4a92aabeee7ec6c73333a929e35"} Oct 30 19:11:56.705552 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:11:56.705359 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1652e2b36293b8c470dd47f23c0f593b59aa4a92aabeee7ec6c73333a929e35" Oct 30 19:12:03.039875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.039839 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt"] Oct 30 19:12:03.040409 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.040374 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0eaffcd7-2ea0-4a5c-ad87-90a65c976251" containerName="revoke-keys" Oct 30 19:12:03.040409 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.040387 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eaffcd7-2ea0-4a5c-ad87-90a65c976251" containerName="revoke-keys" Oct 30 19:12:03.040516 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.040505 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eaffcd7-2ea0-4a5c-ad87-90a65c976251" containerName="revoke-keys" Oct 30 19:12:03.043921 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.043899 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.064441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.064407 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7176ea-rgcwt\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.064586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.064453 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7176ea-rgcwt\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.064644 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.064623 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ncj\" (UniqueName: \"kubernetes.io/projected/ae077ad8-9e07-44f1-86a5-b0c97096e287-kube-api-access-k9ncj\") pod \"maas-api-revoke-sub-af7176ea-rgcwt\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.069539 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.069513 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:03.071612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.069763 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:12:03.073332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.073313 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt"] Oct 30 19:12:03.165813 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.165784 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7176ea-rgcwt\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.165813 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.165816 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7176ea-rgcwt\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.166090 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.165878 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ncj\" (UniqueName: \"kubernetes.io/projected/ae077ad8-9e07-44f1-86a5-b0c97096e287-kube-api-access-k9ncj\") pod \"maas-api-revoke-sub-af7176ea-rgcwt\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.166832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.166803 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af7176ea-rgcwt\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.168447 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.168425 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af7176ea-rgcwt\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.183490 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.183462 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ncj\" (UniqueName: \"kubernetes.io/projected/ae077ad8-9e07-44f1-86a5-b0c97096e287-kube-api-access-k9ncj\") pod \"maas-api-revoke-sub-af7176ea-rgcwt\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.357539 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.357500 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:03.662089 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.662012 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt"] Oct 30 19:12:03.666187 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:12:03.666157 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae077ad8_9e07_44f1_86a5_b0c97096e287.slice/crio-45512345301b5137b97c33a32e24045b34e5b8cfa401a42d49fa8bb5cebe929e WatchSource:0}: Error finding container 45512345301b5137b97c33a32e24045b34e5b8cfa401a42d49fa8bb5cebe929e: Status 404 returned error can't find the container with id 45512345301b5137b97c33a32e24045b34e5b8cfa401a42d49fa8bb5cebe929e Oct 30 19:12:03.753710 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:03.753670 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" event={"ID":"ae077ad8-9e07-44f1-86a5-b0c97096e287","Type":"ContainerStarted","Data":"45512345301b5137b97c33a32e24045b34e5b8cfa401a42d49fa8bb5cebe929e"} Oct 30 19:12:04.761719 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:04.761638 4862 generic.go:356] "Generic (PLEG): container finished" podID="ae077ad8-9e07-44f1-86a5-b0c97096e287" containerID="f805d09776c86c4b63d376ef277373b323d2d791f21e0e3a1705f683c2503de7" exitCode=0 Oct 30 19:12:04.762081 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:04.761728 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" event={"ID":"ae077ad8-9e07-44f1-86a5-b0c97096e287","Type":"ContainerDied","Data":"f805d09776c86c4b63d376ef277373b323d2d791f21e0e3a1705f683c2503de7"} Oct 30 19:12:05.920760 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:05.920736 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:06.014572 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.014543 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-openshift-ca\") pod \"ae077ad8-9e07-44f1-86a5-b0c97096e287\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " Oct 30 19:12:06.014732 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.014659 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-xks-ca\") pod \"ae077ad8-9e07-44f1-86a5-b0c97096e287\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " Oct 30 19:12:06.014806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.014791 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae077ad8-9e07-44f1-86a5-b0c97096e287-kube-api-access-k9ncj\" (UniqueName: \"kubernetes.io/projected/ae077ad8-9e07-44f1-86a5-b0c97096e287-kube-api-access-k9ncj\") pod \"ae077ad8-9e07-44f1-86a5-b0c97096e287\" (UID: \"ae077ad8-9e07-44f1-86a5-b0c97096e287\") " Oct 30 19:12:06.014887 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.014866 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-openshift-ca" pod "ae077ad8-9e07-44f1-86a5-b0c97096e287" (UID: "ae077ad8-9e07-44f1-86a5-b0c97096e287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:06.015230 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.015213 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:06.016854 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.016827 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-xks-ca" pod "ae077ad8-9e07-44f1-86a5-b0c97096e287" (UID: "ae077ad8-9e07-44f1-86a5-b0c97096e287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:06.016948 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.016934 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae077ad8-9e07-44f1-86a5-b0c97096e287-kube-api-access-k9ncj" pod "ae077ad8-9e07-44f1-86a5-b0c97096e287" (UID: "ae077ad8-9e07-44f1-86a5-b0c97096e287"). InnerVolumeSpecName "kube-api-access-k9ncj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:06.116806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.116784 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae077ad8-9e07-44f1-86a5-b0c97096e287-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:06.116806 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.116806 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9ncj\" (UniqueName: \"kubernetes.io/projected/ae077ad8-9e07-44f1-86a5-b0c97096e287-kube-api-access-k9ncj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:06.776275 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.776241 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" Oct 30 19:12:06.776460 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.776273 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt" event={"ID":"ae077ad8-9e07-44f1-86a5-b0c97096e287","Type":"ContainerDied","Data":"45512345301b5137b97c33a32e24045b34e5b8cfa401a42d49fa8bb5cebe929e"} Oct 30 19:12:06.776460 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:06.776326 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45512345301b5137b97c33a32e24045b34e5b8cfa401a42d49fa8bb5cebe929e" Oct 30 19:12:09.339590 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.339561 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-aaf25a"] Oct 30 19:12:09.340055 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.340042 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ae077ad8-9e07-44f1-86a5-b0c97096e287" containerName="revoke-keys" Oct 30 19:12:09.340100 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.340056 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae077ad8-9e07-44f1-86a5-b0c97096e287" containerName="revoke-keys" Oct 30 19:12:09.340179 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.340170 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae077ad8-9e07-44f1-86a5-b0c97096e287" containerName="revoke-keys" Oct 30 19:12:09.343558 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.343539 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-aaf25a" Oct 30 19:12:09.359572 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.359551 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-aaf25a"] Oct 30 19:12:09.366976 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.366954 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r2spd\"" Oct 30 19:12:09.455504 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.455474 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8nl\" (UniqueName: \"kubernetes.io/projected/16cd09a3-3a39-4e41-b516-109b21dca4f4-kube-api-access-pm8nl\") pod \"test-curl-7616-aaf25a\" (UID: \"16cd09a3-3a39-4e41-b516-109b21dca4f4\") " pod="opendatahub/test-curl-7616-aaf25a" Oct 30 19:12:09.556693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.556662 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8nl\" (UniqueName: \"kubernetes.io/projected/16cd09a3-3a39-4e41-b516-109b21dca4f4-kube-api-access-pm8nl\") pod \"test-curl-7616-aaf25a\" (UID: \"16cd09a3-3a39-4e41-b516-109b21dca4f4\") " pod="opendatahub/test-curl-7616-aaf25a" Oct 30 19:12:09.579398 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.579371 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8nl\" (UniqueName: \"kubernetes.io/projected/16cd09a3-3a39-4e41-b516-109b21dca4f4-kube-api-access-pm8nl\") pod \"test-curl-7616-aaf25a\" (UID: \"16cd09a3-3a39-4e41-b516-109b21dca4f4\") " pod="opendatahub/test-curl-7616-aaf25a" Oct 30 19:12:09.654972 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.654917 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-aaf25a" Oct 30 19:12:09.957635 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:09.957441 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-aaf25a"] Oct 30 19:12:10.803575 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:10.803541 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-aaf25a" event={"ID":"16cd09a3-3a39-4e41-b516-109b21dca4f4","Type":"ContainerStarted","Data":"7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1"} Oct 30 19:12:10.803976 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:10.803590 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-aaf25a" event={"ID":"16cd09a3-3a39-4e41-b516-109b21dca4f4","Type":"ContainerStarted","Data":"78feea5d668b474547a8e408522dce50dcf835c2d11fa6f36f9a70e9d044ead7"} Oct 30 19:12:10.836912 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:10.836869 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7616-aaf25a" podStartSLOduration=1.8368571249999999 podStartE2EDuration="1.836857125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:10.836543553 +0000 UTC m=+4429.885588466" watchObservedRunningTime="2026-10-30 19:12:10.836857125 +0000 UTC m=+4429.885902039" Oct 30 19:12:11.708101 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:11.708069 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-aaf25a"] Oct 30 19:12:11.712826 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:11.712801 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-aaf25a"] Oct 30 19:12:11.873347 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:11.873320 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-ac3d93"] Oct 30 19:12:11.873900 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:11.873882 4862 cpu_manager.go:414] "containerMap: removing container" podUID="16cd09a3-3a39-4e41-b516-109b21dca4f4" containerName="test-curl-7616-aaf25a" Oct 30 19:12:11.873948 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:11.873902 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16cd09a3-3a39-4e41-b516-109b21dca4f4" containerName="test-curl-7616-aaf25a" Oct 30 19:12:11.873994 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:11.873982 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="16cd09a3-3a39-4e41-b516-109b21dca4f4" containerName="test-curl-7616-aaf25a" Oct 30 19:12:11.877239 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:11.877223 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-ac3d93" Oct 30 19:12:11.894878 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:11.894856 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-ac3d93"] Oct 30 19:12:11.986496 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:11.986325 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8xf\" (UniqueName: \"kubernetes.io/projected/53260e9d-a243-453c-a921-29dfa91a6c37-kube-api-access-6v8xf\") pod \"test-curl-7616-ac3d93\" (UID: \"53260e9d-a243-453c-a921-29dfa91a6c37\") " pod="opendatahub/test-curl-7616-ac3d93" Oct 30 19:12:12.087906 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:12.087857 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8xf\" (UniqueName: \"kubernetes.io/projected/53260e9d-a243-453c-a921-29dfa91a6c37-kube-api-access-6v8xf\") pod \"test-curl-7616-ac3d93\" (UID: \"53260e9d-a243-453c-a921-29dfa91a6c37\") " pod="opendatahub/test-curl-7616-ac3d93" Oct 30 19:12:12.108431 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:12.108400 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8xf\" (UniqueName: \"kubernetes.io/projected/53260e9d-a243-453c-a921-29dfa91a6c37-kube-api-access-6v8xf\") pod \"test-curl-7616-ac3d93\" (UID: \"53260e9d-a243-453c-a921-29dfa91a6c37\") " pod="opendatahub/test-curl-7616-ac3d93" Oct 30 19:12:12.189371 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:12.189347 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-ac3d93" Oct 30 19:12:12.491391 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:12.491208 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-ac3d93"] Oct 30 19:12:12.494404 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:12:12.494376 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53260e9d_a243_453c_a921_29dfa91a6c37.slice/crio-68bb19654d5c877f5ae5438a967649c5948e10fdeb2364369e06d8a535737f7d WatchSource:0}: Error finding container 68bb19654d5c877f5ae5438a967649c5948e10fdeb2364369e06d8a535737f7d: Status 404 returned error can't find the container with id 68bb19654d5c877f5ae5438a967649c5948e10fdeb2364369e06d8a535737f7d Oct 30 19:12:12.817767 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:12.817733 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-ac3d93" event={"ID":"53260e9d-a243-453c-a921-29dfa91a6c37","Type":"ContainerStarted","Data":"7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96"} Oct 30 19:12:12.817767 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:12.817777 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-ac3d93" event={"ID":"53260e9d-a243-453c-a921-29dfa91a6c37","Type":"ContainerStarted","Data":"68bb19654d5c877f5ae5438a967649c5948e10fdeb2364369e06d8a535737f7d"} Oct 30 19:12:12.818047 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:12.817895 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7616-aaf25a" podUID="16cd09a3-3a39-4e41-b516-109b21dca4f4" containerName="test-curl-7616-aaf25a" containerID="cri-o://7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1" gracePeriod=2 Oct 30 19:12:12.859148 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:12.859105 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7616-ac3d93" podStartSLOduration=1.859092942 podStartE2EDuration="1.859092942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:12.857167112 +0000 UTC m=+4431.906212025" watchObservedRunningTime="2026-10-30 19:12:12.859092942 +0000 UTC m=+4431.908137855" Oct 30 19:12:12.869008 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:12:12.868977 4862 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7616-aaf25a\" is forbidden: User \"system:node:ip-10-0-128-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-246.ec2.internal' and this object" podUID="16cd09a3-3a39-4e41-b516-109b21dca4f4" pod="opendatahub/test-curl-7616-aaf25a" Oct 30 19:12:12.880966 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:12:12.880938 4862 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7616-aaf25a\" is forbidden: User \"system:node:ip-10-0-128-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-246.ec2.internal' and this object" podUID="16cd09a3-3a39-4e41-b516-109b21dca4f4" pod="opendatahub/test-curl-7616-aaf25a" Oct 30 19:12:13.705986 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:13.705954 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-ac3d93"] Oct 30 19:12:13.710153 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:13.710125 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-ac3d93"] Oct 30 19:12:13.914000 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:13.913970 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2"] Oct 30 19:12:14.416587 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.416546 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc"] Oct 30 19:12:14.417065 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.417051 4862 cpu_manager.go:414] "containerMap: removing container" podUID="53260e9d-a243-453c-a921-29dfa91a6c37" containerName="test-curl-7616-ac3d93" Oct 30 19:12:14.417137 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.417065 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53260e9d-a243-453c-a921-29dfa91a6c37" containerName="test-curl-7616-ac3d93" Oct 30 19:12:14.417195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.417178 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="53260e9d-a243-453c-a921-29dfa91a6c37" containerName="test-curl-7616-ac3d93" Oct 30 19:12:14.420590 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.420565 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.438339 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.438317 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc"] Oct 30 19:12:14.509115 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.509080 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:14.509302 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.509178 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:12:14.517397 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.517371 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009f69e1-pt2vc\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.517597 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.517426 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qjl\" (UniqueName: \"kubernetes.io/projected/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-kube-api-access-69qjl\") pod \"maas-api-revoke-sub-009f69e1-pt2vc\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.517597 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.517533 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009f69e1-pt2vc\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.618767 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.618731 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009f69e1-pt2vc\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.618962 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.618796 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qjl\" (UniqueName: \"kubernetes.io/projected/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-kube-api-access-69qjl\") pod \"maas-api-revoke-sub-009f69e1-pt2vc\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.618962 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.618855 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009f69e1-pt2vc\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.619481 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.619459 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-009f69e1-pt2vc\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.621460 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.621439 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-009f69e1-pt2vc\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.636871 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.636847 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qjl\" (UniqueName: \"kubernetes.io/projected/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-kube-api-access-69qjl\") pod \"maas-api-revoke-sub-009f69e1-pt2vc\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.734441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.734311 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:14.831247 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:14.831217 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7616-ac3d93" podUID="53260e9d-a243-453c-a921-29dfa91a6c37" containerName="test-curl-7616-ac3d93" containerID="cri-o://7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96" gracePeriod=2 Oct 30 19:12:14.843144 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:12:14.843115 4862 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7616-ac3d93\" is forbidden: User \"system:node:ip-10-0-128-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-246.ec2.internal' and this object" podUID="53260e9d-a243-453c-a921-29dfa91a6c37" pod="opendatahub/test-curl-7616-ac3d93" Oct 30 19:12:15.010792 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.010767 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-aaf25a" Oct 30 19:12:15.022890 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:12:15.022859 4862 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7616-ac3d93\" is forbidden: User \"system:node:ip-10-0-128-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-246.ec2.internal' and this object" podUID="53260e9d-a243-453c-a921-29dfa91a6c37" pod="opendatahub/test-curl-7616-ac3d93" Oct 30 19:12:15.042616 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.042595 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc"] Oct 30 19:12:15.124951 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.124924 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16cd09a3-3a39-4e41-b516-109b21dca4f4-kube-api-access-pm8nl\" (UniqueName: \"kubernetes.io/projected/16cd09a3-3a39-4e41-b516-109b21dca4f4-kube-api-access-pm8nl\") pod \"16cd09a3-3a39-4e41-b516-109b21dca4f4\" (UID: \"16cd09a3-3a39-4e41-b516-109b21dca4f4\") " Oct 30 19:12:15.127033 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.127011 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cd09a3-3a39-4e41-b516-109b21dca4f4-kube-api-access-pm8nl" pod "16cd09a3-3a39-4e41-b516-109b21dca4f4" (UID: "16cd09a3-3a39-4e41-b516-109b21dca4f4"). InnerVolumeSpecName "kube-api-access-pm8nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:15.227161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.227136 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm8nl\" (UniqueName: \"kubernetes.io/projected/16cd09a3-3a39-4e41-b516-109b21dca4f4-kube-api-access-pm8nl\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:15.839902 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.839870 4862 generic.go:356] "Generic (PLEG): container finished" podID="16cd09a3-3a39-4e41-b516-109b21dca4f4" containerID="7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1" exitCode=137 Oct 30 19:12:15.840086 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.839913 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-aaf25a" Oct 30 19:12:15.840086 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.839970 4862 scope.go:122] "RemoveContainer" containerID="7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1" Oct 30 19:12:15.841938 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.841909 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" event={"ID":"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c","Type":"ContainerStarted","Data":"5182b50ac5f9b2bee3123df5afe3b3d98aaeaa659a72c5b4b64d4135ea87c9a6"} Oct 30 19:12:15.842058 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.841972 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" event={"ID":"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c","Type":"ContainerStarted","Data":"710fb947f98064d72cab281b33d25260f5f931367a9656250a85b6a6f6a6c181"} Oct 30 19:12:15.850226 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.850210 4862 scope.go:122] "RemoveContainer" containerID="7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1" Oct 30 19:12:15.850574 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:12:15.850554 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1\": container with ID starting with 7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1 not found: ID does not exist" containerID="7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1" Oct 30 19:12:15.850642 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.850582 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1"} err="failed to get container status \"7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1\": rpc error: code = NotFound desc = could not find container \"7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1\": container with ID starting with 7c73287fda69f9b690fc58f607728c9bba023ae72a2ad5ca99706557b05d4df1 not found: ID does not exist" Oct 30 19:12:15.877443 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:15.877405 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" podStartSLOduration=1.87739449 podStartE2EDuration="1.87739449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:15.876789863 +0000 UTC m=+4434.925834775" watchObservedRunningTime="2026-10-30 19:12:15.87739449 +0000 UTC m=+4434.926439442" Oct 30 19:12:16.852735 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:16.852705 4862 generic.go:356] "Generic (PLEG): container finished" podID="49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c" containerID="5182b50ac5f9b2bee3123df5afe3b3d98aaeaa659a72c5b4b64d4135ea87c9a6" exitCode=0 Oct 30 19:12:16.853085 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:16.852789 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" event={"ID":"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c","Type":"ContainerDied","Data":"5182b50ac5f9b2bee3123df5afe3b3d98aaeaa659a72c5b4b64d4135ea87c9a6"} Oct 30 19:12:17.050602 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.050580 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-ac3d93" Oct 30 19:12:17.061972 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.061949 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16cd09a3-3a39-4e41-b516-109b21dca4f4" path="/var/lib/kubelet/pods/16cd09a3-3a39-4e41-b516-109b21dca4f4/volumes" Oct 30 19:12:17.148820 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.148763 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53260e9d-a243-453c-a921-29dfa91a6c37-kube-api-access-6v8xf\" (UniqueName: \"kubernetes.io/projected/53260e9d-a243-453c-a921-29dfa91a6c37-kube-api-access-6v8xf\") pod \"53260e9d-a243-453c-a921-29dfa91a6c37\" (UID: \"53260e9d-a243-453c-a921-29dfa91a6c37\") " Oct 30 19:12:17.151089 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.151065 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53260e9d-a243-453c-a921-29dfa91a6c37-kube-api-access-6v8xf" pod "53260e9d-a243-453c-a921-29dfa91a6c37" (UID: "53260e9d-a243-453c-a921-29dfa91a6c37"). InnerVolumeSpecName "kube-api-access-6v8xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:17.250211 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.250182 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8xf\" (UniqueName: \"kubernetes.io/projected/53260e9d-a243-453c-a921-29dfa91a6c37-kube-api-access-6v8xf\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:17.861810 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.861776 4862 generic.go:356] "Generic (PLEG): container finished" podID="53260e9d-a243-453c-a921-29dfa91a6c37" containerID="7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96" exitCode=137 Oct 30 19:12:17.862278 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.862046 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-ac3d93" Oct 30 19:12:17.862278 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.862088 4862 scope.go:122] "RemoveContainer" containerID="7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96" Oct 30 19:12:17.880778 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.880753 4862 scope.go:122] "RemoveContainer" containerID="7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96" Oct 30 19:12:17.881151 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:12:17.881120 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96\": container with ID starting with 7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96 not found: ID does not exist" containerID="7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96" Oct 30 19:12:17.881299 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:17.881163 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96"} err="failed to get container status \"7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96\": rpc error: code = NotFound desc = could not find container \"7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96\": container with ID starting with 7520bc04fb5d1a26857d1160f4d9423713159015850267087fb4d080b3cf5f96 not found: ID does not exist" Oct 30 19:12:18.012363 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.012341 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:18.160923 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.160842 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-openshift-ca\") pod \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " Oct 30 19:12:18.160923 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.160879 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-xks-ca\") pod \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " Oct 30 19:12:18.160923 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.160909 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-kube-api-access-69qjl\" (UniqueName: \"kubernetes.io/projected/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-kube-api-access-69qjl\") pod \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\" (UID: \"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c\") " Oct 30 19:12:18.161294 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.161235 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-openshift-ca" pod "49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c" (UID: "49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:18.161593 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.161569 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:18.163419 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.163391 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-xks-ca" pod "49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c" (UID: "49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:18.163649 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.163626 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-kube-api-access-69qjl" pod "49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c" (UID: "49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c"). InnerVolumeSpecName "kube-api-access-69qjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:18.262497 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.262473 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:18.262497 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.262496 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69qjl\" (UniqueName: \"kubernetes.io/projected/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c-kube-api-access-69qjl\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:18.876122 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.876088 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" Oct 30 19:12:18.877649 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.876094 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc" event={"ID":"49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c","Type":"ContainerDied","Data":"710fb947f98064d72cab281b33d25260f5f931367a9656250a85b6a6f6a6c181"} Oct 30 19:12:18.877649 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:18.877605 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710fb947f98064d72cab281b33d25260f5f931367a9656250a85b6a6f6a6c181" Oct 30 19:12:19.062936 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:19.062903 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53260e9d-a243-453c-a921-29dfa91a6c37" path="/var/lib/kubelet/pods/53260e9d-a243-453c-a921-29dfa91a6c37/volumes" Oct 30 19:12:19.876613 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:12:19.876547 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:12:23.200056 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.200023 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:12:23.855174 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.855126 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64"] Oct 30 19:12:23.855873 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.855848 4862 cpu_manager.go:414] "containerMap: removing container" podUID="49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c" containerName="revoke-keys" Oct 30 19:12:23.855873 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.855870 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c" containerName="revoke-keys" Oct 30 19:12:23.856060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.855991 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c" containerName="revoke-keys" Oct 30 19:12:23.860616 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.860594 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:23.880378 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.879471 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64"] Oct 30 19:12:23.890283 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.886478 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:23.890283 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.886563 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:12:23.969357 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.969324 4862 scope.go:122] "RemoveContainer" containerID="9a5a83355fd406dce1f196bb14b4be6fcb160328a70e2c6f53feee4b19a64f5d" Oct 30 19:12:23.983459 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:23.983436 4862 scope.go:122] "RemoveContainer" containerID="d37781471f11269e1bbca8e34694c0d43e9da3df4ef741054fef315a75cd9914" Oct 30 19:12:24.035858 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.035832 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.035972 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.035872 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvs59\" (UniqueName: \"kubernetes.io/projected/9da8c60a-cd98-40b8-bf79-b0828261589d-kube-api-access-bvs59\") pod \"maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.036141 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.036116 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.137022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.136944 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.137022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.136996 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvs59\" (UniqueName: \"kubernetes.io/projected/9da8c60a-cd98-40b8-bf79-b0828261589d-kube-api-access-bvs59\") pod \"maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.137280 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.137227 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.137668 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.137637 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.139913 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.139890 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.154635 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.154609 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvs59\" (UniqueName: \"kubernetes.io/projected/9da8c60a-cd98-40b8-bf79-b0828261589d-kube-api-access-bvs59\") pod \"maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.184775 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.184751 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:24.505382 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.505357 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64"] Oct 30 19:12:24.507623 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:12:24.507596 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da8c60a_cd98_40b8_bf79_b0828261589d.slice/crio-7f9644ed840b14c8611985f9b083273ae54a50289add66931e801abba3cd097f WatchSource:0}: Error finding container 7f9644ed840b14c8611985f9b083273ae54a50289add66931e801abba3cd097f: Status 404 returned error can't find the container with id 7f9644ed840b14c8611985f9b083273ae54a50289add66931e801abba3cd097f Oct 30 19:12:24.993288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:24.992732 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" event={"ID":"9da8c60a-cd98-40b8-bf79-b0828261589d","Type":"ContainerStarted","Data":"7f9644ed840b14c8611985f9b083273ae54a50289add66931e801abba3cd097f"} Oct 30 19:12:26.011012 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:26.010919 4862 generic.go:356] "Generic (PLEG): container finished" podID="9da8c60a-cd98-40b8-bf79-b0828261589d" containerID="272a4bffff8045ea16797b805d4da36a1a3560c2a29bd66aac43061ebd3a5f12" exitCode=0 Oct 30 19:12:26.011425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:26.010999 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" event={"ID":"9da8c60a-cd98-40b8-bf79-b0828261589d","Type":"ContainerDied","Data":"272a4bffff8045ea16797b805d4da36a1a3560c2a29bd66aac43061ebd3a5f12"} Oct 30 19:12:27.181082 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.181057 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:27.201173 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.201149 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da8c60a-cd98-40b8-bf79-b0828261589d-kube-api-access-bvs59\" (UniqueName: \"kubernetes.io/projected/9da8c60a-cd98-40b8-bf79-b0828261589d-kube-api-access-bvs59\") pod \"9da8c60a-cd98-40b8-bf79-b0828261589d\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " Oct 30 19:12:27.201331 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.201272 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-xks-ca\") pod \"9da8c60a-cd98-40b8-bf79-b0828261589d\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " Oct 30 19:12:27.201386 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.201332 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-openshift-ca\") pod \"9da8c60a-cd98-40b8-bf79-b0828261589d\" (UID: \"9da8c60a-cd98-40b8-bf79-b0828261589d\") " Oct 30 19:12:27.201709 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.201683 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-openshift-ca" pod "9da8c60a-cd98-40b8-bf79-b0828261589d" (UID: "9da8c60a-cd98-40b8-bf79-b0828261589d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:27.203538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.203514 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-xks-ca" pod "9da8c60a-cd98-40b8-bf79-b0828261589d" (UID: "9da8c60a-cd98-40b8-bf79-b0828261589d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:27.203626 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.203529 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da8c60a-cd98-40b8-bf79-b0828261589d-kube-api-access-bvs59" pod "9da8c60a-cd98-40b8-bf79-b0828261589d" (UID: "9da8c60a-cd98-40b8-bf79-b0828261589d"). InnerVolumeSpecName "kube-api-access-bvs59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:27.302553 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.302525 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvs59\" (UniqueName: \"kubernetes.io/projected/9da8c60a-cd98-40b8-bf79-b0828261589d-kube-api-access-bvs59\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:27.302553 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.302550 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:27.302737 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:27.302559 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da8c60a-cd98-40b8-bf79-b0828261589d-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:28.026517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:28.026486 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" event={"ID":"9da8c60a-cd98-40b8-bf79-b0828261589d","Type":"ContainerDied","Data":"7f9644ed840b14c8611985f9b083273ae54a50289add66931e801abba3cd097f"} Oct 30 19:12:28.026517 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:28.026517 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64" Oct 30 19:12:28.026730 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:28.026536 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f9644ed840b14c8611985f9b083273ae54a50289add66931e801abba3cd097f" Oct 30 19:12:28.939637 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:28.939592 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" podUID="b7508ead-c54b-4b59-a49b-46df7370cac2" containerName="main" containerID="cri-o://0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820" gracePeriod=45 Oct 30 19:12:29.205735 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.205682 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:12:29.224955 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.224932 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7508ead-c54b-4b59-a49b-46df7370cac2-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7508ead-c54b-4b59-a49b-46df7370cac2-tls-certs\") pod \"b7508ead-c54b-4b59-a49b-46df7370cac2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " Oct 30 19:12:29.225067 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.224980 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-model-cache\") pod \"b7508ead-c54b-4b59-a49b-46df7370cac2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " Oct 30 19:12:29.225067 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225017 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-dshm\") pod \"b7508ead-c54b-4b59-a49b-46df7370cac2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " Oct 30 19:12:29.225177 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225078 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-home\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-home\") pod \"b7508ead-c54b-4b59-a49b-46df7370cac2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " Oct 30 19:12:29.225177 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225096 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-tmp-dir\") pod \"b7508ead-c54b-4b59-a49b-46df7370cac2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " Oct 30 19:12:29.225177 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225171 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7508ead-c54b-4b59-a49b-46df7370cac2-kube-api-access-vvnql\" (UniqueName: \"kubernetes.io/projected/b7508ead-c54b-4b59-a49b-46df7370cac2-kube-api-access-vvnql\") pod \"b7508ead-c54b-4b59-a49b-46df7370cac2\" (UID: \"b7508ead-c54b-4b59-a49b-46df7370cac2\") " Oct 30 19:12:29.225362 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225257 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-model-cache" pod "b7508ead-c54b-4b59-a49b-46df7370cac2" (UID: "b7508ead-c54b-4b59-a49b-46df7370cac2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:29.225438 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225411 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-tmp-dir" pod "b7508ead-c54b-4b59-a49b-46df7370cac2" (UID: "b7508ead-c54b-4b59-a49b-46df7370cac2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:29.225438 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225421 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-home" pod "b7508ead-c54b-4b59-a49b-46df7370cac2" (UID: "b7508ead-c54b-4b59-a49b-46df7370cac2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:29.225764 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225742 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:29.226019 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225766 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:29.226019 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.225780 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:29.227488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.227462 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-dshm" pod "b7508ead-c54b-4b59-a49b-46df7370cac2" (UID: "b7508ead-c54b-4b59-a49b-46df7370cac2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:12:29.227587 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.227498 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7508ead-c54b-4b59-a49b-46df7370cac2-kube-api-access-vvnql" pod "b7508ead-c54b-4b59-a49b-46df7370cac2" (UID: "b7508ead-c54b-4b59-a49b-46df7370cac2"). InnerVolumeSpecName "kube-api-access-vvnql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:29.227587 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.227516 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7508ead-c54b-4b59-a49b-46df7370cac2-tls-certs" pod "b7508ead-c54b-4b59-a49b-46df7370cac2" (UID: "b7508ead-c54b-4b59-a49b-46df7370cac2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:29.327240 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.327209 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvnql\" (UniqueName: \"kubernetes.io/projected/b7508ead-c54b-4b59-a49b-46df7370cac2-kube-api-access-vvnql\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:29.327240 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.327243 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7508ead-c54b-4b59-a49b-46df7370cac2-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:29.327453 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:29.327256 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7508ead-c54b-4b59-a49b-46df7370cac2-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:30.040538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:30.040498 4862 generic.go:356] "Generic (PLEG): container finished" podID="b7508ead-c54b-4b59-a49b-46df7370cac2" containerID="0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820" exitCode=0 Oct 30 19:12:30.041034 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:30.040542 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" Oct 30 19:12:30.041034 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:30.040618 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" event={"ID":"b7508ead-c54b-4b59-a49b-46df7370cac2","Type":"ContainerDied","Data":"0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820"} Oct 30 19:12:30.041034 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:30.040664 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2" event={"ID":"b7508ead-c54b-4b59-a49b-46df7370cac2","Type":"ContainerDied","Data":"dbc46a85534efddf01d4b5f4eca938f8ac31c873a3061cbe4fbd1d4a07353323"} Oct 30 19:12:30.041034 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:30.040687 4862 scope.go:122] "RemoveContainer" containerID="0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820" Oct 30 19:12:30.051521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:30.051500 4862 scope.go:122] "RemoveContainer" containerID="0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820" Oct 30 19:12:30.051831 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:12:30.051812 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820\": container with ID starting with 0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820 not found: ID does not exist" containerID="0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820" Oct 30 19:12:30.051900 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:30.051843 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820"} err="failed to get container status \"0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820\": rpc error: code = NotFound desc = could not find container \"0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820\": container with ID starting with 0a3d5791b7bbe54aa5868904cbf6822826b11a051a65a2a8c8ebd87a8d126820 not found: ID does not exist" Oct 30 19:12:30.092561 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:30.092527 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2"] Oct 30 19:12:30.102336 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:30.102307 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1a2436/rate-test-model-1a2436-kserve-6f45766669-49vp2"] Oct 30 19:12:31.071961 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.071929 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7508ead-c54b-4b59-a49b-46df7370cac2" path="/var/lib/kubelet/pods/b7508ead-c54b-4b59-a49b-46df7370cac2/volumes" Oct 30 19:12:31.152249 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.152224 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff"] Oct 30 19:12:31.152717 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.152704 4862 cpu_manager.go:414] "containerMap: removing container" podUID="9da8c60a-cd98-40b8-bf79-b0828261589d" containerName="revoke-keys" Oct 30 19:12:31.152766 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.152718 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da8c60a-cd98-40b8-bf79-b0828261589d" containerName="revoke-keys" Oct 30 19:12:31.152766 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.152737 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b7508ead-c54b-4b59-a49b-46df7370cac2" containerName="main" Oct 30 19:12:31.152766 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.152742 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7508ead-c54b-4b59-a49b-46df7370cac2" containerName="main" Oct 30 19:12:31.152938 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.152861 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da8c60a-cd98-40b8-bf79-b0828261589d" containerName="revoke-keys" Oct 30 19:12:31.152938 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.152872 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7508ead-c54b-4b59-a49b-46df7370cac2" containerName="main" Oct 30 19:12:31.157727 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.157683 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.174460 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.174434 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff"] Oct 30 19:12:31.181028 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.181005 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:12:31.181842 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.181424 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:31.248803 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.248720 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ndj\" (UniqueName: \"kubernetes.io/projected/bf1700b0-10fd-4357-96b4-f665910f6ebb-kube-api-access-q2ndj\") pod \"maas-api-revoke-sub-d9f7c159-6q4ff\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.248803 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.248784 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9f7c159-6q4ff\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.249055 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.249031 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9f7c159-6q4ff\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.350180 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.350109 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9f7c159-6q4ff\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.350385 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.350190 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ndj\" (UniqueName: \"kubernetes.io/projected/bf1700b0-10fd-4357-96b4-f665910f6ebb-kube-api-access-q2ndj\") pod \"maas-api-revoke-sub-d9f7c159-6q4ff\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.350385 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.350234 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9f7c159-6q4ff\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.350900 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.350880 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9f7c159-6q4ff\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.352645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.352625 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9f7c159-6q4ff\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.367958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.367934 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ndj\" (UniqueName: \"kubernetes.io/projected/bf1700b0-10fd-4357-96b4-f665910f6ebb-kube-api-access-q2ndj\") pod \"maas-api-revoke-sub-d9f7c159-6q4ff\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.476013 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.475992 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:31.778876 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:31.778853 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff"] Oct 30 19:12:31.781191 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:12:31.781164 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1700b0_10fd_4357_96b4_f665910f6ebb.slice/crio-483deb0d366ff331f3c6d28bbe5eb829c6ff94710a4f591037d7fb6ea5498729 WatchSource:0}: Error finding container 483deb0d366ff331f3c6d28bbe5eb829c6ff94710a4f591037d7fb6ea5498729: Status 404 returned error can't find the container with id 483deb0d366ff331f3c6d28bbe5eb829c6ff94710a4f591037d7fb6ea5498729 Oct 30 19:12:32.057496 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:32.057469 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" event={"ID":"bf1700b0-10fd-4357-96b4-f665910f6ebb","Type":"ContainerStarted","Data":"8e75315d5af8d44989a9dd2e246300d27950b4058e0108356fc3900e6afa03de"} Oct 30 19:12:32.057618 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:32.057524 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" event={"ID":"bf1700b0-10fd-4357-96b4-f665910f6ebb","Type":"ContainerStarted","Data":"483deb0d366ff331f3c6d28bbe5eb829c6ff94710a4f591037d7fb6ea5498729"} Oct 30 19:12:32.093394 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:32.093348 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" podStartSLOduration=1.093331565 podStartE2EDuration="1.093331565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:32.092766687 +0000 UTC m=+4451.141811601" watchObservedRunningTime="2026-10-30 19:12:32.093331565 +0000 UTC m=+4451.142376492" Oct 30 19:12:33.065413 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:33.065384 4862 generic.go:356] "Generic (PLEG): container finished" podID="bf1700b0-10fd-4357-96b4-f665910f6ebb" containerID="8e75315d5af8d44989a9dd2e246300d27950b4058e0108356fc3900e6afa03de" exitCode=0 Oct 30 19:12:33.065617 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:33.065476 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" event={"ID":"bf1700b0-10fd-4357-96b4-f665910f6ebb","Type":"ContainerDied","Data":"8e75315d5af8d44989a9dd2e246300d27950b4058e0108356fc3900e6afa03de"} Oct 30 19:12:34.221216 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.221192 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:34.283109 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.283079 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-xks-ca\") pod \"bf1700b0-10fd-4357-96b4-f665910f6ebb\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " Oct 30 19:12:34.283288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.283143 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1700b0-10fd-4357-96b4-f665910f6ebb-kube-api-access-q2ndj\" (UniqueName: \"kubernetes.io/projected/bf1700b0-10fd-4357-96b4-f665910f6ebb-kube-api-access-q2ndj\") pod \"bf1700b0-10fd-4357-96b4-f665910f6ebb\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " Oct 30 19:12:34.283369 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.283332 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-openshift-ca\") pod \"bf1700b0-10fd-4357-96b4-f665910f6ebb\" (UID: \"bf1700b0-10fd-4357-96b4-f665910f6ebb\") " Oct 30 19:12:34.283754 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.283724 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-openshift-ca" pod "bf1700b0-10fd-4357-96b4-f665910f6ebb" (UID: "bf1700b0-10fd-4357-96b4-f665910f6ebb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:34.285429 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.285400 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-xks-ca" pod "bf1700b0-10fd-4357-96b4-f665910f6ebb" (UID: "bf1700b0-10fd-4357-96b4-f665910f6ebb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:34.285429 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.285407 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1700b0-10fd-4357-96b4-f665910f6ebb-kube-api-access-q2ndj" pod "bf1700b0-10fd-4357-96b4-f665910f6ebb" (UID: "bf1700b0-10fd-4357-96b4-f665910f6ebb"). InnerVolumeSpecName "kube-api-access-q2ndj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:34.384637 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.384579 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:34.384637 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.384607 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ndj\" (UniqueName: \"kubernetes.io/projected/bf1700b0-10fd-4357-96b4-f665910f6ebb-kube-api-access-q2ndj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:34.384637 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:34.384623 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1700b0-10fd-4357-96b4-f665910f6ebb-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:35.079798 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.079775 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" Oct 30 19:12:35.079957 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.079814 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff" event={"ID":"bf1700b0-10fd-4357-96b4-f665910f6ebb","Type":"ContainerDied","Data":"483deb0d366ff331f3c6d28bbe5eb829c6ff94710a4f591037d7fb6ea5498729"} Oct 30 19:12:35.079957 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.079864 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="483deb0d366ff331f3c6d28bbe5eb829c6ff94710a4f591037d7fb6ea5498729" Oct 30 19:12:35.735572 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.735542 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m"] Oct 30 19:12:35.736087 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.736063 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bf1700b0-10fd-4357-96b4-f665910f6ebb" containerName="revoke-keys" Oct 30 19:12:35.736087 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.736079 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1700b0-10fd-4357-96b4-f665910f6ebb" containerName="revoke-keys" Oct 30 19:12:35.736216 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.736166 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1700b0-10fd-4357-96b4-f665910f6ebb" containerName="revoke-keys" Oct 30 19:12:35.739598 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.739580 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:35.760224 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.760187 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m"] Oct 30 19:12:35.782797 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.782773 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:35.785356 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.785340 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:12:35.804707 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.804681 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrg9\" (UniqueName: \"kubernetes.io/projected/62015e36-11c0-4b1e-ba85-d76c99717575-kube-api-access-bmrg9\") pod \"maas-api-revoke-sub-5f7418fe-5mk5m\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:35.804811 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.804790 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f7418fe-5mk5m\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:35.804918 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.804900 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f7418fe-5mk5m\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:35.905656 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.905624 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f7418fe-5mk5m\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:35.905790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.905744 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f7418fe-5mk5m\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:35.905867 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.905851 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrg9\" (UniqueName: \"kubernetes.io/projected/62015e36-11c0-4b1e-ba85-d76c99717575-kube-api-access-bmrg9\") pod \"maas-api-revoke-sub-5f7418fe-5mk5m\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:35.906345 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.906323 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f7418fe-5mk5m\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:35.908089 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.908064 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f7418fe-5mk5m\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:35.923857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:35.923836 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrg9\" (UniqueName: \"kubernetes.io/projected/62015e36-11c0-4b1e-ba85-d76c99717575-kube-api-access-bmrg9\") pod \"maas-api-revoke-sub-5f7418fe-5mk5m\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:36.053676 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:36.053655 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:36.354761 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:36.354733 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m"] Oct 30 19:12:36.357305 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:12:36.357278 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62015e36_11c0_4b1e_ba85_d76c99717575.slice/crio-094d4fd421de3c9561080c76986964d9ea7cb3d4bd9e742ac20fb674cc4b8c0e WatchSource:0}: Error finding container 094d4fd421de3c9561080c76986964d9ea7cb3d4bd9e742ac20fb674cc4b8c0e: Status 404 returned error can't find the container with id 094d4fd421de3c9561080c76986964d9ea7cb3d4bd9e742ac20fb674cc4b8c0e Oct 30 19:12:37.095612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.095577 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" event={"ID":"62015e36-11c0-4b1e-ba85-d76c99717575","Type":"ContainerStarted","Data":"80497bd29d0549b587857f13613665910dc5a657ac61eb8744457ff4efb6f247"} Oct 30 19:12:37.096016 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.095631 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" event={"ID":"62015e36-11c0-4b1e-ba85-d76c99717575","Type":"ContainerStarted","Data":"094d4fd421de3c9561080c76986964d9ea7cb3d4bd9e742ac20fb674cc4b8c0e"} Oct 30 19:12:37.134119 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.134071 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" podStartSLOduration=2.134058509 podStartE2EDuration="2.134058509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:37.13110381 +0000 UTC m=+4456.180148722" watchObservedRunningTime="2026-10-30 19:12:37.134058509 +0000 UTC m=+4456.183103461" Oct 30 19:12:37.170502 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.170476 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh"] Oct 30 19:12:37.175046 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.175026 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.189146 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.189128 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-c095fa-openshift-default-dockercfg-kvfnl\"" Oct 30 19:12:37.205773 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.205744 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh"] Oct 30 19:12:37.219812 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.219787 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-podinfo\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.219961 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.219899 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7da89a5-2e98-4151-a29a-f350338eca2d-istiod-ca-cert\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.219961 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.219940 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxtbn\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-kube-api-access-sxtbn\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.220091 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.219977 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-certs\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.220091 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.220064 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-credential-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.220206 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.220114 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.220206 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.220153 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-token\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.220206 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.220197 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-envoy\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.220410 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.220243 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-data\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321326 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321289 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-certs\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321367 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-credential-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321413 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321488 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321448 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-token\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321671 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321487 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-envoy\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321671 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321533 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-data\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321671 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321624 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-podinfo\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321836 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321698 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-certs\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321836 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321726 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7da89a5-2e98-4151-a29a-f350338eca2d-istiod-ca-cert\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321836 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321750 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-credential-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321836 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321766 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxtbn\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-kube-api-access-sxtbn\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.321836 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.321829 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-socket\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.322301 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.322277 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-data\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.322576 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.322329 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7da89a5-2e98-4151-a29a-f350338eca2d-istiod-ca-cert\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.323914 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.323887 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-envoy\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.324166 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.324149 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-podinfo\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.339543 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.339517 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-token\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.339641 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.339562 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxtbn\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-kube-api-access-sxtbn\") pod \"e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.489129 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.489102 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:37.806029 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.805995 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh"] Oct 30 19:12:37.807577 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:12:37.807546 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7da89a5_2e98_4151_a29a_f350338eca2d.slice/crio-e6012321ed93675eedecd86fef3406ac5a139176bd028d6543c5d237c5edfcba WatchSource:0}: Error finding container e6012321ed93675eedecd86fef3406ac5a139176bd028d6543c5d237c5edfcba: Status 404 returned error can't find the container with id e6012321ed93675eedecd86fef3406ac5a139176bd028d6543c5d237c5edfcba Oct 30 19:12:37.810380 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.810347 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:12:37.810475 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.810414 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:12:37.810475 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:37.810445 4862 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:12:38.104930 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:38.104835 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" event={"ID":"e7da89a5-2e98-4151-a29a-f350338eca2d","Type":"ContainerStarted","Data":"8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6"} Oct 30 19:12:38.104930 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:38.104917 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" event={"ID":"e7da89a5-2e98-4151-a29a-f350338eca2d","Type":"ContainerStarted","Data":"e6012321ed93675eedecd86fef3406ac5a139176bd028d6543c5d237c5edfcba"} Oct 30 19:12:38.106701 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:38.106677 4862 generic.go:356] "Generic (PLEG): container finished" podID="62015e36-11c0-4b1e-ba85-d76c99717575" containerID="80497bd29d0549b587857f13613665910dc5a657ac61eb8744457ff4efb6f247" exitCode=0 Oct 30 19:12:38.106832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:38.106762 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" event={"ID":"62015e36-11c0-4b1e-ba85-d76c99717575","Type":"ContainerDied","Data":"80497bd29d0549b587857f13613665910dc5a657ac61eb8744457ff4efb6f247"} Oct 30 19:12:38.147678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:38.147623 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" podStartSLOduration=1.14760581 podStartE2EDuration="1.14760581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:38.144727237 +0000 UTC m=+4457.193772150" watchObservedRunningTime="2026-10-30 19:12:38.14760581 +0000 UTC m=+4457.196650727" Oct 30 19:12:38.489910 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:38.489837 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:38.491249 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:38.491224 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 19:12:38.491388 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:38.491302 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 19:12:39.312385 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.312359 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:39.472321 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.472238 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62015e36-11c0-4b1e-ba85-d76c99717575-kube-api-access-bmrg9\" (UniqueName: \"kubernetes.io/projected/62015e36-11c0-4b1e-ba85-d76c99717575-kube-api-access-bmrg9\") pod \"62015e36-11c0-4b1e-ba85-d76c99717575\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " Oct 30 19:12:39.472485 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.472376 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-openshift-ca\") pod \"62015e36-11c0-4b1e-ba85-d76c99717575\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " Oct 30 19:12:39.472485 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.472436 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-xks-ca\") pod \"62015e36-11c0-4b1e-ba85-d76c99717575\" (UID: \"62015e36-11c0-4b1e-ba85-d76c99717575\") " Oct 30 19:12:39.472807 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.472784 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-openshift-ca" pod "62015e36-11c0-4b1e-ba85-d76c99717575" (UID: "62015e36-11c0-4b1e-ba85-d76c99717575"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:39.472961 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.472946 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:39.474535 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.474512 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62015e36-11c0-4b1e-ba85-d76c99717575-kube-api-access-bmrg9" pod "62015e36-11c0-4b1e-ba85-d76c99717575" (UID: "62015e36-11c0-4b1e-ba85-d76c99717575"). InnerVolumeSpecName "kube-api-access-bmrg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:39.474613 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.474575 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-xks-ca" pod "62015e36-11c0-4b1e-ba85-d76c99717575" (UID: "62015e36-11c0-4b1e-ba85-d76c99717575"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:39.489927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.489894 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 19:12:39.490027 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.489944 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 19:12:39.574074 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.574048 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmrg9\" (UniqueName: \"kubernetes.io/projected/62015e36-11c0-4b1e-ba85-d76c99717575-kube-api-access-bmrg9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:39.574074 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:39.574074 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62015e36-11c0-4b1e-ba85-d76c99717575-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:40.121275 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:40.121227 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" Oct 30 19:12:40.121468 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:40.121295 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m" event={"ID":"62015e36-11c0-4b1e-ba85-d76c99717575","Type":"ContainerDied","Data":"094d4fd421de3c9561080c76986964d9ea7cb3d4bd9e742ac20fb674cc4b8c0e"} Oct 30 19:12:40.121468 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:40.121345 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094d4fd421de3c9561080c76986964d9ea7cb3d4bd9e742ac20fb674cc4b8c0e" Oct 30 19:12:40.489830 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:40.489752 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 19:12:40.489830 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:40.489806 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 19:12:41.489984 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.489957 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 19:12:41.490417 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.490008 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 19:12:41.555568 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.555524 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5"] Oct 30 19:12:41.556133 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.556115 4862 cpu_manager.go:414] "containerMap: removing container" podUID="62015e36-11c0-4b1e-ba85-d76c99717575" containerName="revoke-keys" Oct 30 19:12:41.556133 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.556132 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62015e36-11c0-4b1e-ba85-d76c99717575" containerName="revoke-keys" Oct 30 19:12:41.556366 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.556255 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="62015e36-11c0-4b1e-ba85-d76c99717575" containerName="revoke-keys" Oct 30 19:12:41.559637 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.559618 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.566639 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.566604 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5"] Oct 30 19:12:41.572810 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.572790 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:12:41.573605 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.573587 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:41.699560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.699525 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv6m8\" (UniqueName: \"kubernetes.io/projected/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-kube-api-access-fv6m8\") pod \"maas-api-revoke-sub-b1120e8d-d2bf5\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.699737 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.699580 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1120e8d-d2bf5\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.699786 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.699731 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1120e8d-d2bf5\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.800877 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.800843 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1120e8d-d2bf5\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.801044 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.800956 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv6m8\" (UniqueName: \"kubernetes.io/projected/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-kube-api-access-fv6m8\") pod \"maas-api-revoke-sub-b1120e8d-d2bf5\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.801044 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.800998 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1120e8d-d2bf5\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.801640 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.801614 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1120e8d-d2bf5\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.803544 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.803524 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1120e8d-d2bf5\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.819538 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.819515 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv6m8\" (UniqueName: \"kubernetes.io/projected/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-kube-api-access-fv6m8\") pod \"maas-api-revoke-sub-b1120e8d-d2bf5\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:41.876017 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:41.875987 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:42.200409 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:42.200368 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5"] Oct 30 19:12:42.202713 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:12:42.202692 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65b1b3b_b94c_457c_ab38_712fa4ad7d13.slice/crio-6db5d1ca9001610b6d7be7a867430ce5e4b53cd240315bcdf04cb18da422b6a6 WatchSource:0}: Error finding container 6db5d1ca9001610b6d7be7a867430ce5e4b53cd240315bcdf04cb18da422b6a6: Status 404 returned error can't find the container with id 6db5d1ca9001610b6d7be7a867430ce5e4b53cd240315bcdf04cb18da422b6a6 Oct 30 19:12:42.490161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:42.490134 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 19:12:42.490506 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:42.490183 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 19:12:43.169601 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:43.169555 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" event={"ID":"d65b1b3b-b94c-457c-ab38-712fa4ad7d13","Type":"ContainerStarted","Data":"6e572f256d0dbcbb0984d08211daee4a42f74ba68bdfe3268bcad2d3ecce6eda"} Oct 30 19:12:43.169809 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:43.169624 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" event={"ID":"d65b1b3b-b94c-457c-ab38-712fa4ad7d13","Type":"ContainerStarted","Data":"6db5d1ca9001610b6d7be7a867430ce5e4b53cd240315bcdf04cb18da422b6a6"} Oct 30 19:12:43.206074 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:43.206025 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" podStartSLOduration=2.206009611 podStartE2EDuration="2.206009611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:43.204156344 +0000 UTC m=+4462.253201258" watchObservedRunningTime="2026-10-30 19:12:43.206009611 +0000 UTC m=+4462.255054555" Oct 30 19:12:43.490213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:43.489876 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 19:12:43.490213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:43.489934 4862 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 19:12:44.193516 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:44.192759 4862 generic.go:356] "Generic (PLEG): container finished" podID="d65b1b3b-b94c-457c-ab38-712fa4ad7d13" containerID="6e572f256d0dbcbb0984d08211daee4a42f74ba68bdfe3268bcad2d3ecce6eda" exitCode=0 Oct 30 19:12:44.193516 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:44.192999 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" event={"ID":"d65b1b3b-b94c-457c-ab38-712fa4ad7d13","Type":"ContainerDied","Data":"6e572f256d0dbcbb0984d08211daee4a42f74ba68bdfe3268bcad2d3ecce6eda"} Oct 30 19:12:44.647825 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:44.647787 4862 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:44.648418 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:44.648391 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:44.649470 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:44.649444 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:12:45.366709 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.366683 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:45.470896 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.470793 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-xks-ca\") pod \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " Oct 30 19:12:45.470896 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.470867 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-openshift-ca\") pod \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " Oct 30 19:12:45.471136 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.470975 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-kube-api-access-fv6m8\" (UniqueName: \"kubernetes.io/projected/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-kube-api-access-fv6m8\") pod \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\" (UID: \"d65b1b3b-b94c-457c-ab38-712fa4ad7d13\") " Oct 30 19:12:45.471788 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.471461 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-openshift-ca" pod "d65b1b3b-b94c-457c-ab38-712fa4ad7d13" (UID: "d65b1b3b-b94c-457c-ab38-712fa4ad7d13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:45.471788 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.471621 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:45.473419 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.473387 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-xks-ca" pod "d65b1b3b-b94c-457c-ab38-712fa4ad7d13" (UID: "d65b1b3b-b94c-457c-ab38-712fa4ad7d13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:45.474033 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.474011 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-kube-api-access-fv6m8" pod "d65b1b3b-b94c-457c-ab38-712fa4ad7d13" (UID: "d65b1b3b-b94c-457c-ab38-712fa4ad7d13"). InnerVolumeSpecName "kube-api-access-fv6m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:45.572773 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.572736 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:45.572966 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:45.572767 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv6m8\" (UniqueName: \"kubernetes.io/projected/d65b1b3b-b94c-457c-ab38-712fa4ad7d13-kube-api-access-fv6m8\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:46.208560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.208531 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" Oct 30 19:12:46.209055 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.208556 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5" event={"ID":"d65b1b3b-b94c-457c-ab38-712fa4ad7d13","Type":"ContainerDied","Data":"6db5d1ca9001610b6d7be7a867430ce5e4b53cd240315bcdf04cb18da422b6a6"} Oct 30 19:12:46.209055 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.208602 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db5d1ca9001610b6d7be7a867430ce5e4b53cd240315bcdf04cb18da422b6a6" Oct 30 19:12:46.352737 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.352709 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt"] Oct 30 19:12:46.353347 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.353326 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d65b1b3b-b94c-457c-ab38-712fa4ad7d13" containerName="revoke-keys" Oct 30 19:12:46.353347 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.353347 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65b1b3b-b94c-457c-ab38-712fa4ad7d13" containerName="revoke-keys" Oct 30 19:12:46.353532 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.353477 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65b1b3b-b94c-457c-ab38-712fa4ad7d13" containerName="revoke-keys" Oct 30 19:12:46.356990 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.356971 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.371812 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.371787 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:12:46.374346 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.374311 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:12:46.375794 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.375774 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt"] Oct 30 19:12:46.484860 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.484781 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpj7q\" (UniqueName: \"kubernetes.io/projected/bbb9bd28-af5e-40a7-985f-509d0bded58c-kube-api-access-lpj7q\") pod \"maas-api-revoke-sub-0a615556-ccrzt\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.485002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.484898 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a615556-ccrzt\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.485002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.484960 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a615556-ccrzt\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.586634 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.586601 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpj7q\" (UniqueName: \"kubernetes.io/projected/bbb9bd28-af5e-40a7-985f-509d0bded58c-kube-api-access-lpj7q\") pod \"maas-api-revoke-sub-0a615556-ccrzt\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.586829 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.586675 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a615556-ccrzt\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.586829 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.586712 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a615556-ccrzt\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.587327 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.587308 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a615556-ccrzt\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.589224 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.589201 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a615556-ccrzt\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.607239 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.607214 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpj7q\" (UniqueName: \"kubernetes.io/projected/bbb9bd28-af5e-40a7-985f-509d0bded58c-kube-api-access-lpj7q\") pod \"maas-api-revoke-sub-0a615556-ccrzt\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.670793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.670771 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:46.977676 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:46.977651 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt"] Oct 30 19:12:46.980621 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:12:46.980595 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb9bd28_af5e_40a7_985f_509d0bded58c.slice/crio-898ad2fa1169480f1fbf637a2818f215a0506920198ad2a27b0224dcaf4afffb WatchSource:0}: Error finding container 898ad2fa1169480f1fbf637a2818f215a0506920198ad2a27b0224dcaf4afffb: Status 404 returned error can't find the container with id 898ad2fa1169480f1fbf637a2818f215a0506920198ad2a27b0224dcaf4afffb Oct 30 19:12:47.215742 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:47.215713 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" event={"ID":"bbb9bd28-af5e-40a7-985f-509d0bded58c","Type":"ContainerStarted","Data":"898ad2fa1169480f1fbf637a2818f215a0506920198ad2a27b0224dcaf4afffb"} Oct 30 19:12:48.223984 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:48.223906 4862 generic.go:356] "Generic (PLEG): container finished" podID="bbb9bd28-af5e-40a7-985f-509d0bded58c" containerID="4dc5729bd179061909f5b87d403ae1c9bb4c309fb7722df0820e7f86d61d9877" exitCode=0 Oct 30 19:12:48.224477 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:48.223995 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" event={"ID":"bbb9bd28-af5e-40a7-985f-509d0bded58c","Type":"ContainerDied","Data":"4dc5729bd179061909f5b87d403ae1c9bb4c309fb7722df0820e7f86d61d9877"} Oct 30 19:12:49.393978 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.393945 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:49.522324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.522220 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-xks-ca\") pod \"bbb9bd28-af5e-40a7-985f-509d0bded58c\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " Oct 30 19:12:49.522496 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.522335 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb9bd28-af5e-40a7-985f-509d0bded58c-kube-api-access-lpj7q\" (UniqueName: \"kubernetes.io/projected/bbb9bd28-af5e-40a7-985f-509d0bded58c-kube-api-access-lpj7q\") pod \"bbb9bd28-af5e-40a7-985f-509d0bded58c\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " Oct 30 19:12:49.522496 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.522442 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-openshift-ca\") pod \"bbb9bd28-af5e-40a7-985f-509d0bded58c\" (UID: \"bbb9bd28-af5e-40a7-985f-509d0bded58c\") " Oct 30 19:12:49.522817 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.522792 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-openshift-ca" pod "bbb9bd28-af5e-40a7-985f-509d0bded58c" (UID: "bbb9bd28-af5e-40a7-985f-509d0bded58c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:12:49.523005 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.522985 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:49.524437 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.524412 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-xks-ca" pod "bbb9bd28-af5e-40a7-985f-509d0bded58c" (UID: "bbb9bd28-af5e-40a7-985f-509d0bded58c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:12:49.524612 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.524590 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb9bd28-af5e-40a7-985f-509d0bded58c-kube-api-access-lpj7q" pod "bbb9bd28-af5e-40a7-985f-509d0bded58c" (UID: "bbb9bd28-af5e-40a7-985f-509d0bded58c"). InnerVolumeSpecName "kube-api-access-lpj7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:12:49.625096 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.625072 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb9bd28-af5e-40a7-985f-509d0bded58c-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:49.625096 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:49.625104 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpj7q\" (UniqueName: \"kubernetes.io/projected/bbb9bd28-af5e-40a7-985f-509d0bded58c-kube-api-access-lpj7q\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:12:49.870490 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:12:49.870453 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:12:50.240935 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:50.240851 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" event={"ID":"bbb9bd28-af5e-40a7-985f-509d0bded58c","Type":"ContainerDied","Data":"898ad2fa1169480f1fbf637a2818f215a0506920198ad2a27b0224dcaf4afffb"} Oct 30 19:12:50.240935 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:50.240878 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt" Oct 30 19:12:50.240935 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:12:50.240912 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898ad2fa1169480f1fbf637a2818f215a0506920198ad2a27b0224dcaf4afffb" Oct 30 19:13:02.125469 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:02.125434 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64"] Oct 30 19:13:02.135760 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:02.135725 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1c4bcd-jnf64"] Oct 30 19:13:03.062323 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:03.062285 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da8c60a-cd98-40b8-bf79-b0828261589d" path="/var/lib/kubelet/pods/9da8c60a-cd98-40b8-bf79-b0828261589d/volumes" Oct 30 19:13:04.272720 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.272689 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp"] Oct 30 19:13:04.548113 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.548077 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj"] Oct 30 19:13:04.548618 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.548605 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bbb9bd28-af5e-40a7-985f-509d0bded58c" containerName="revoke-keys" Oct 30 19:13:04.548683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.548620 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb9bd28-af5e-40a7-985f-509d0bded58c" containerName="revoke-keys" Oct 30 19:13:04.548759 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.548750 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb9bd28-af5e-40a7-985f-509d0bded58c" containerName="revoke-keys" Oct 30 19:13:04.552129 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.552112 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.580962 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.580938 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj"] Oct 30 19:13:04.597693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.597659 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.597841 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.597730 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.597841 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.597766 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bq8r\" (UniqueName: \"kubernetes.io/projected/028d1f6f-f6e8-43bc-a753-ded273901271-kube-api-access-2bq8r\") pod \"maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.601327 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.601303 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:04.601456 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.601346 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:13:04.691978 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.691940 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:13:04.698930 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.698909 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.699028 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.698950 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.699028 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.698973 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bq8r\" (UniqueName: \"kubernetes.io/projected/028d1f6f-f6e8-43bc-a753-ded273901271-kube-api-access-2bq8r\") pod \"maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.699607 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.699578 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.701547 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.701519 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.720166 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.720145 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bq8r\" (UniqueName: \"kubernetes.io/projected/028d1f6f-f6e8-43bc-a753-ded273901271-kube-api-access-2bq8r\") pod \"maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:04.866617 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:04.866551 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:05.170463 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:05.170439 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj"] Oct 30 19:13:05.173056 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:13:05.173033 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028d1f6f_f6e8_43bc_a753_ded273901271.slice/crio-b015e797fd928460d1e4a0942d9c59da5c5fab779cb71653cd86c2f1750db78e WatchSource:0}: Error finding container b015e797fd928460d1e4a0942d9c59da5c5fab779cb71653cd86c2f1750db78e: Status 404 returned error can't find the container with id b015e797fd928460d1e4a0942d9c59da5c5fab779cb71653cd86c2f1750db78e Oct 30 19:13:05.344691 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:05.344654 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" event={"ID":"028d1f6f-f6e8-43bc-a753-ded273901271","Type":"ContainerStarted","Data":"b015e797fd928460d1e4a0942d9c59da5c5fab779cb71653cd86c2f1750db78e"} Oct 30 19:13:06.352939 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:06.352904 4862 generic.go:356] "Generic (PLEG): container finished" podID="028d1f6f-f6e8-43bc-a753-ded273901271" containerID="8639a384528c9c0128a87a163a6eeb8e4120c6e9102c6c9d777b5d7ee0efeff2" exitCode=0 Oct 30 19:13:06.353393 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:06.352988 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" event={"ID":"028d1f6f-f6e8-43bc-a753-ded273901271","Type":"ContainerDied","Data":"8639a384528c9c0128a87a163a6eeb8e4120c6e9102c6c9d777b5d7ee0efeff2"} Oct 30 19:13:07.511116 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.511094 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:07.635535 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.635459 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-xks-ca\") pod \"028d1f6f-f6e8-43bc-a753-ded273901271\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " Oct 30 19:13:07.635535 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.635495 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028d1f6f-f6e8-43bc-a753-ded273901271-kube-api-access-2bq8r\" (UniqueName: \"kubernetes.io/projected/028d1f6f-f6e8-43bc-a753-ded273901271-kube-api-access-2bq8r\") pod \"028d1f6f-f6e8-43bc-a753-ded273901271\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " Oct 30 19:13:07.635744 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.635628 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-openshift-ca\") pod \"028d1f6f-f6e8-43bc-a753-ded273901271\" (UID: \"028d1f6f-f6e8-43bc-a753-ded273901271\") " Oct 30 19:13:07.636065 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.636036 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-openshift-ca" pod "028d1f6f-f6e8-43bc-a753-ded273901271" (UID: "028d1f6f-f6e8-43bc-a753-ded273901271"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:07.637872 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.637842 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-xks-ca" pod "028d1f6f-f6e8-43bc-a753-ded273901271" (UID: "028d1f6f-f6e8-43bc-a753-ded273901271"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:07.638023 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.637960 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028d1f6f-f6e8-43bc-a753-ded273901271-kube-api-access-2bq8r" pod "028d1f6f-f6e8-43bc-a753-ded273901271" (UID: "028d1f6f-f6e8-43bc-a753-ded273901271"). InnerVolumeSpecName "kube-api-access-2bq8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:07.737210 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.737184 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:07.737210 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.737209 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bq8r\" (UniqueName: \"kubernetes.io/projected/028d1f6f-f6e8-43bc-a753-ded273901271-kube-api-access-2bq8r\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:07.737441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:07.737222 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028d1f6f-f6e8-43bc-a753-ded273901271-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:08.366682 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:08.366654 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" Oct 30 19:13:08.366849 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:08.366690 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj" event={"ID":"028d1f6f-f6e8-43bc-a753-ded273901271","Type":"ContainerDied","Data":"b015e797fd928460d1e4a0942d9c59da5c5fab779cb71653cd86c2f1750db78e"} Oct 30 19:13:08.366849 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:08.366739 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b015e797fd928460d1e4a0942d9c59da5c5fab779cb71653cd86c2f1750db78e" Oct 30 19:13:14.733134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:14.733093 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs"] Oct 30 19:13:19.294327 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.294254 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" podUID="ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" containerName="main" containerID="cri-o://7e3a330c6828d224d51c84d1907bb67f15697dd07ee7c1d5b997d6b5ed7d67db" gracePeriod=45 Oct 30 19:13:19.444856 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.444823 4862 generic.go:356] "Generic (PLEG): container finished" podID="ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" containerID="7e3a330c6828d224d51c84d1907bb67f15697dd07ee7c1d5b997d6b5ed7d67db" exitCode=0 Oct 30 19:13:19.445046 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.444902 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" event={"ID":"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729","Type":"ContainerDied","Data":"7e3a330c6828d224d51c84d1907bb67f15697dd07ee7c1d5b997d6b5ed7d67db"} Oct 30 19:13:19.619790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.619765 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:13:19.733953 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.733914 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" podUID="8345433c-4916-4e21-af2a-9a5214ed504d" containerName="payload-pre-processing" containerID="cri-o://3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6" gracePeriod=25 Oct 30 19:13:19.807434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.807409 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-home\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-home\") pod \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " Oct 30 19:13:19.807589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.807456 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-dshm\") pod \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " Oct 30 19:13:19.807589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.807497 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-kube-api-access-tbc48\" (UniqueName: \"kubernetes.io/projected/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-kube-api-access-tbc48\") pod \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " Oct 30 19:13:19.807589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.807530 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-model-cache\") pod \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " Oct 30 19:13:19.807589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.807579 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tmp-dir\") pod \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " Oct 30 19:13:19.807828 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.807640 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tls-certs\") pod \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\" (UID: \"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729\") " Oct 30 19:13:19.807828 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.807641 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-home" pod "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" (UID: "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:19.807828 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.807790 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-model-cache" pod "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" (UID: "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:19.807984 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.807924 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tmp-dir" pod "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" (UID: "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:19.808151 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.808131 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.808197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.808158 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.808197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.808172 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.809999 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.809981 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-dshm" pod "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" (UID: "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:19.810072 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.809996 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tls-certs" pod "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" (UID: "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:19.810072 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.810023 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-kube-api-access-tbc48" pod "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" (UID: "ef7feae8-dfc9-4e75-b95e-e1a29a5f1729"). InnerVolumeSpecName "kube-api-access-tbc48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:19.876685 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:13:19.876588 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:13:19.909560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.909527 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.909560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.909557 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbc48\" (UniqueName: \"kubernetes.io/projected/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-kube-api-access-tbc48\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:19.909758 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:19.909567 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:20.454319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:20.454282 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn"] Oct 30 19:13:20.454786 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:20.454645 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" Oct 30 19:13:20.454786 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:20.454666 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp" event={"ID":"ef7feae8-dfc9-4e75-b95e-e1a29a5f1729","Type":"ContainerDied","Data":"e95d7377c35e4971cc576bbbbf2a757c5dffe683f4e740b1ce83d71721a7d7f4"} Oct 30 19:13:20.454786 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:20.454706 4862 scope.go:122] "RemoveContainer" containerID="7e3a330c6828d224d51c84d1907bb67f15697dd07ee7c1d5b997d6b5ed7d67db" Oct 30 19:13:20.527354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:20.527323 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp"] Oct 30 19:13:20.528999 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:20.528976 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c095fa/sub-test-model-c095fa-kserve-7c9cd56c8f-dfvcp"] Oct 30 19:13:21.062172 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:21.062143 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" path="/var/lib/kubelet/pods/ef7feae8-dfc9-4e75-b95e-e1a29a5f1729/volumes" Oct 30 19:13:22.060573 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:22.060541 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 19:13:22.068639 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:22.068619 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 19:13:24.033928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:24.033901 4862 scope.go:122] "RemoveContainer" containerID="a55a1ad183beae5d53aa5a097952b65931e9a346a34d5aedd21fe8ad23db92cf" Oct 30 19:13:24.048057 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:24.048038 4862 scope.go:122] "RemoveContainer" containerID="82087453256bb3371d5132060b66523770680df73c1ee43d2fa828bab8a95eff" Oct 30 19:13:24.057195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:24.057175 4862 scope.go:122] "RemoveContainer" containerID="d32fd5e2a66c854a9862792b2f6c285ff5d7e3ba3afc2c0f91742d2781e36d43" Oct 30 19:13:25.455433 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:25.455394 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" podUID="a6df5a42-1fc7-4159-b61a-ab0c969e5985" containerName="payload-pre-processing" containerID="cri-o://1b764ef0fde7e2805147f265c8af7576856f2cc19b870f6a90ff7c3a33c16ffd" gracePeriod=25 Oct 30 19:13:27.615157 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:27.615124 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj"] Oct 30 19:13:27.624737 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:27.624714 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-713dc8-rzvmj"] Oct 30 19:13:29.062473 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:29.062441 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028d1f6f-f6e8-43bc-a753-ded273901271" path="/var/lib/kubelet/pods/028d1f6f-f6e8-43bc-a753-ded273901271/volumes" Oct 30 19:13:29.174560 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:29.174525 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" podUID="8345433c-4916-4e21-af2a-9a5214ed504d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:33.281368 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:33.281328 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" podUID="a6df5a42-1fc7-4159-b61a-ab0c969e5985" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:34.886111 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:34.886075 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:13:38.288990 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.288955 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5"] Oct 30 19:13:38.354338 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.354305 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd"] Oct 30 19:13:38.354752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.354717 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" podUID="f8dc3f79-83b1-4e35-92f4-6a13a39834e3" containerName="maas-api" containerID="cri-o://ce415efe65901e698fee487b49e39f644004399e6967c6a72205044786af653b" gracePeriod=30 Oct 30 19:13:38.582900 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.582859 4862 generic.go:356] "Generic (PLEG): container finished" podID="f8dc3f79-83b1-4e35-92f4-6a13a39834e3" containerID="ce415efe65901e698fee487b49e39f644004399e6967c6a72205044786af653b" exitCode=0 Oct 30 19:13:38.583079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.582901 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" event={"ID":"f8dc3f79-83b1-4e35-92f4-6a13a39834e3","Type":"ContainerDied","Data":"ce415efe65901e698fee487b49e39f644004399e6967c6a72205044786af653b"} Oct 30 19:13:38.653979 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.653631 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 19:13:38.721943 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.721905 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-maas-api-tls\") pod \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " Oct 30 19:13:38.722134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.721970 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-openshift-service-ca\") pod \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " Oct 30 19:13:38.722134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.722112 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-kube-api-access-hgbqm\" (UniqueName: \"kubernetes.io/projected/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-kube-api-access-hgbqm\") pod \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " Oct 30 19:13:38.722297 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.722193 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-metrics-certs\") pod \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\" (UID: \"f8dc3f79-83b1-4e35-92f4-6a13a39834e3\") " Oct 30 19:13:38.722370 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.722342 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-openshift-service-ca" pod "f8dc3f79-83b1-4e35-92f4-6a13a39834e3" (UID: "f8dc3f79-83b1-4e35-92f4-6a13a39834e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:38.723213 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.723187 4862 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-openshift-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:38.724771 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.724736 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-metrics-certs" pod "f8dc3f79-83b1-4e35-92f4-6a13a39834e3" (UID: "f8dc3f79-83b1-4e35-92f4-6a13a39834e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:38.725022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.725003 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-kube-api-access-hgbqm" pod "f8dc3f79-83b1-4e35-92f4-6a13a39834e3" (UID: "f8dc3f79-83b1-4e35-92f4-6a13a39834e3"). InnerVolumeSpecName "kube-api-access-hgbqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:38.725105 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.725002 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-maas-api-tls" pod "f8dc3f79-83b1-4e35-92f4-6a13a39834e3" (UID: "f8dc3f79-83b1-4e35-92f4-6a13a39834e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:13:38.824819 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.824791 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgbqm\" (UniqueName: \"kubernetes.io/projected/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-kube-api-access-hgbqm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:38.824819 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.824818 4862 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-metrics-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:38.824819 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:38.824829 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8dc3f79-83b1-4e35-92f4-6a13a39834e3-maas-api-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:39.174691 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:39.174652 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" podUID="8345433c-4916-4e21-af2a-9a5214ed504d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:39.591148 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:39.591114 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" Oct 30 19:13:39.591148 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:39.591139 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" event={"ID":"f8dc3f79-83b1-4e35-92f4-6a13a39834e3","Type":"ContainerDied","Data":"2b7027cfabdf8d73254247bd7c0d72823b381cd6a7f6266c8b45af7665c86fd6"} Oct 30 19:13:39.591720 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:39.591196 4862 scope.go:122] "RemoveContainer" containerID="ce415efe65901e698fee487b49e39f644004399e6967c6a72205044786af653b" Oct 30 19:13:39.638249 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:39.638226 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd"] Oct 30 19:13:39.649852 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:39.649828 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd"] Oct 30 19:13:40.008067 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.008039 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 19:13:40.161071 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.160992 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8345433c-4916-4e21-af2a-9a5214ed504d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8345433c-4916-4e21-af2a-9a5214ed504d-plugins-config-volume\") pod \"8345433c-4916-4e21-af2a-9a5214ed504d\" (UID: \"8345433c-4916-4e21-af2a-9a5214ed504d\") " Oct 30 19:13:40.161248 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.161211 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8345433c-4916-4e21-af2a-9a5214ed504d-kube-api-access-h6f68\" (UniqueName: \"kubernetes.io/projected/8345433c-4916-4e21-af2a-9a5214ed504d-kube-api-access-h6f68\") pod \"8345433c-4916-4e21-af2a-9a5214ed504d\" (UID: \"8345433c-4916-4e21-af2a-9a5214ed504d\") " Oct 30 19:13:40.161457 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.161437 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8345433c-4916-4e21-af2a-9a5214ed504d-plugins-config-volume" pod "8345433c-4916-4e21-af2a-9a5214ed504d" (UID: "8345433c-4916-4e21-af2a-9a5214ed504d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:40.161833 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.161810 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8345433c-4916-4e21-af2a-9a5214ed504d-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.163462 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.163441 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8345433c-4916-4e21-af2a-9a5214ed504d-kube-api-access-h6f68" pod "8345433c-4916-4e21-af2a-9a5214ed504d" (UID: "8345433c-4916-4e21-af2a-9a5214ed504d"). InnerVolumeSpecName "kube-api-access-h6f68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:40.262672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.262639 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6f68\" (UniqueName: \"kubernetes.io/projected/8345433c-4916-4e21-af2a-9a5214ed504d-kube-api-access-h6f68\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:40.599112 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.599082 4862 generic.go:356] "Generic (PLEG): container finished" podID="8345433c-4916-4e21-af2a-9a5214ed504d" containerID="3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6" exitCode=0 Oct 30 19:13:40.599546 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.599125 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" Oct 30 19:13:40.599546 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.599170 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" event={"ID":"8345433c-4916-4e21-af2a-9a5214ed504d","Type":"ContainerDied","Data":"3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6"} Oct 30 19:13:40.599546 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.599219 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs" event={"ID":"8345433c-4916-4e21-af2a-9a5214ed504d","Type":"ContainerDied","Data":"494f2b1ff6be32b4c45424b6fb4151920ab34dfe61aa6f42e46e8e54ea6e7f97"} Oct 30 19:13:40.599546 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.599246 4862 scope.go:122] "RemoveContainer" containerID="3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6" Oct 30 19:13:40.610324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.610307 4862 scope.go:122] "RemoveContainer" containerID="3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6" Oct 30 19:13:40.610582 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:13:40.610559 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6\": container with ID starting with 3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6 not found: ID does not exist" containerID="3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6" Oct 30 19:13:40.610648 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.610593 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6"} err="failed to get container status \"3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6\": rpc error: code = NotFound desc = could not find container \"3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6\": container with ID starting with 3259e28c723a6d3d400216b83ddedcd54e0b6ba0544e8d397ce718177029dee6 not found: ID does not exist" Oct 30 19:13:40.651180 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.651156 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs"] Oct 30 19:13:40.661425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:40.661403 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-713dc8-798785c495-lqlhs"] Oct 30 19:13:41.063164 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:41.063132 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8345433c-4916-4e21-af2a-9a5214ed504d" path="/var/lib/kubelet/pods/8345433c-4916-4e21-af2a-9a5214ed504d/volumes" Oct 30 19:13:41.063691 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:41.063669 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8dc3f79-83b1-4e35-92f4-6a13a39834e3" path="/var/lib/kubelet/pods/f8dc3f79-83b1-4e35-92f4-6a13a39834e3/volumes" Oct 30 19:13:41.564484 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:41.564426 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-8b8e54-7968547d9d-pbndd" podUID="f8dc3f79-83b1-4e35-92f4-6a13a39834e3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.117:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:13:43.281482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:43.281440 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" podUID="a6df5a42-1fc7-4159-b61a-ab0c969e5985" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:43.290194 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:43.290165 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" podUID="b33431f9-a4e7-4691-9fbb-1226b6483fab" containerName="payload-pre-processing" containerID="cri-o://9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59" gracePeriod=25 Oct 30 19:13:45.660784 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:45.660753 4862 generic.go:356] "Generic (PLEG): container finished" podID="a6df5a42-1fc7-4159-b61a-ab0c969e5985" containerID="1b764ef0fde7e2805147f265c8af7576856f2cc19b870f6a90ff7c3a33c16ffd" exitCode=0 Oct 30 19:13:45.661297 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:45.660800 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" event={"ID":"a6df5a42-1fc7-4159-b61a-ab0c969e5985","Type":"ContainerDied","Data":"1b764ef0fde7e2805147f265c8af7576856f2cc19b870f6a90ff7c3a33c16ffd"} Oct 30 19:13:45.731919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:45.731886 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 19:13:45.822113 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:45.822074 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6df5a42-1fc7-4159-b61a-ab0c969e5985-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6df5a42-1fc7-4159-b61a-ab0c969e5985-plugins-config-volume\") pod \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\" (UID: \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\") " Oct 30 19:13:45.822340 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:45.822192 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6df5a42-1fc7-4159-b61a-ab0c969e5985-kube-api-access-p84cc\" (UniqueName: \"kubernetes.io/projected/a6df5a42-1fc7-4159-b61a-ab0c969e5985-kube-api-access-p84cc\") pod \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\" (UID: \"a6df5a42-1fc7-4159-b61a-ab0c969e5985\") " Oct 30 19:13:45.822563 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:45.822536 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6df5a42-1fc7-4159-b61a-ab0c969e5985-plugins-config-volume" pod "a6df5a42-1fc7-4159-b61a-ab0c969e5985" (UID: "a6df5a42-1fc7-4159-b61a-ab0c969e5985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:45.824588 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:45.824560 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6df5a42-1fc7-4159-b61a-ab0c969e5985-kube-api-access-p84cc" pod "a6df5a42-1fc7-4159-b61a-ab0c969e5985" (UID: "a6df5a42-1fc7-4159-b61a-ab0c969e5985"). InnerVolumeSpecName "kube-api-access-p84cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:45.923335 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:45.923303 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6df5a42-1fc7-4159-b61a-ab0c969e5985-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:45.923335 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:45.923334 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p84cc\" (UniqueName: \"kubernetes.io/projected/a6df5a42-1fc7-4159-b61a-ab0c969e5985-kube-api-access-p84cc\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:46.105853 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:46.105818 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2"] Oct 30 19:13:46.507193 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:46.507111 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" podUID="b33431f9-a4e7-4691-9fbb-1226b6483fab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:46.668762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:46.668726 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" event={"ID":"a6df5a42-1fc7-4159-b61a-ab0c969e5985","Type":"ContainerDied","Data":"609c6a61f9ab462e7ee9507169bb3340c3ce72ac33fdefe208ce78912a594060"} Oct 30 19:13:46.669297 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:46.668784 4862 scope.go:122] "RemoveContainer" containerID="1b764ef0fde7e2805147f265c8af7576856f2cc19b870f6a90ff7c3a33c16ffd" Oct 30 19:13:46.669297 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:46.668734 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn" Oct 30 19:13:46.725046 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:46.725013 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn"] Oct 30 19:13:46.727690 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:46.727669 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8b8e54-7b688ffbbc-nl5nn"] Oct 30 19:13:47.061284 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:47.061235 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6df5a42-1fc7-4159-b61a-ab0c969e5985" path="/var/lib/kubelet/pods/a6df5a42-1fc7-4159-b61a-ab0c969e5985/volumes" Oct 30 19:13:47.336115 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:47.336034 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm"] Oct 30 19:13:47.336300 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:47.336240 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" containerID="cri-o://5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c" gracePeriod=30 Oct 30 19:13:49.869010 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:13:49.868964 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:13:50.697785 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:50.697754 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:13:51.107098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.107066 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" podUID="0e4bee85-2a79-4470-a777-dc2ebb574684" containerName="payload-processing" containerID="cri-o://09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58" gracePeriod=25 Oct 30 19:13:51.378911 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.378881 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 19:13:51.500189 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.500159 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e4bee85-2a79-4470-a777-dc2ebb574684-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4bee85-2a79-4470-a777-dc2ebb574684-plugins-config-volume\") pod \"0e4bee85-2a79-4470-a777-dc2ebb574684\" (UID: \"0e4bee85-2a79-4470-a777-dc2ebb574684\") " Oct 30 19:13:51.500404 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.500229 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e4bee85-2a79-4470-a777-dc2ebb574684-kube-api-access-7xp82\" (UniqueName: \"kubernetes.io/projected/0e4bee85-2a79-4470-a777-dc2ebb574684-kube-api-access-7xp82\") pod \"0e4bee85-2a79-4470-a777-dc2ebb574684\" (UID: \"0e4bee85-2a79-4470-a777-dc2ebb574684\") " Oct 30 19:13:51.500636 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.500604 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4bee85-2a79-4470-a777-dc2ebb574684-plugins-config-volume" pod "0e4bee85-2a79-4470-a777-dc2ebb574684" (UID: "0e4bee85-2a79-4470-a777-dc2ebb574684"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:51.500873 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.500855 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4bee85-2a79-4470-a777-dc2ebb574684-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:51.502601 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.502572 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4bee85-2a79-4470-a777-dc2ebb574684-kube-api-access-7xp82" pod "0e4bee85-2a79-4470-a777-dc2ebb574684" (UID: "0e4bee85-2a79-4470-a777-dc2ebb574684"). InnerVolumeSpecName "kube-api-access-7xp82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:51.603325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.603247 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xp82\" (UniqueName: \"kubernetes.io/projected/0e4bee85-2a79-4470-a777-dc2ebb574684-kube-api-access-7xp82\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:51.707161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.707075 4862 generic.go:356] "Generic (PLEG): container finished" podID="0e4bee85-2a79-4470-a777-dc2ebb574684" containerID="09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58" exitCode=0 Oct 30 19:13:51.707161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.707115 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" Oct 30 19:13:51.707436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.707160 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" event={"ID":"0e4bee85-2a79-4470-a777-dc2ebb574684","Type":"ContainerDied","Data":"09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58"} Oct 30 19:13:51.707436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.707211 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2" event={"ID":"0e4bee85-2a79-4470-a777-dc2ebb574684","Type":"ContainerDied","Data":"c213e8b4094bbd4ed206a3de949a043025a53e4625fab7f62df5d899a1597d87"} Oct 30 19:13:51.707436 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.707231 4862 scope.go:122] "RemoveContainer" containerID="09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58" Oct 30 19:13:51.719894 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.719872 4862 scope.go:122] "RemoveContainer" containerID="09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58" Oct 30 19:13:51.720167 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:13:51.720145 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58\": container with ID starting with 09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58 not found: ID does not exist" containerID="09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58" Oct 30 19:13:51.720290 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.720172 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58"} err="failed to get container status \"09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58\": rpc error: code = NotFound desc = could not find container \"09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58\": container with ID starting with 09e73069cea5179d925863c92cad72f3cdbba9be82063dfa47a5fd74bcd52b58 not found: ID does not exist" Oct 30 19:13:51.756404 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.756376 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2"] Oct 30 19:13:51.761861 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:51.761836 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-39c99d-57d5954c5b-cv4v2"] Oct 30 19:13:52.627494 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.627470 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 19:13:52.724551 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.724519 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-podinfo\") pod \"bebbe758-a515-41cd-a91d-05d4647b74e5\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " Oct 30 19:13:52.724750 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.724598 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-socket\") pod \"bebbe758-a515-41cd-a91d-05d4647b74e5\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " Oct 30 19:13:52.724750 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.724629 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-kube-api-access-r6gn5\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-kube-api-access-r6gn5\") pod \"bebbe758-a515-41cd-a91d-05d4647b74e5\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " Oct 30 19:13:52.724750 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.724665 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bebbe758-a515-41cd-a91d-05d4647b74e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebbe758-a515-41cd-a91d-05d4647b74e5-istiod-ca-cert\") pod \"bebbe758-a515-41cd-a91d-05d4647b74e5\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " Oct 30 19:13:52.724958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.724928 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-socket" pod "bebbe758-a515-41cd-a91d-05d4647b74e5" (UID: "bebbe758-a515-41cd-a91d-05d4647b74e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:52.725115 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.725086 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-envoy\") pod \"bebbe758-a515-41cd-a91d-05d4647b74e5\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " Oct 30 19:13:52.725214 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.725154 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebbe758-a515-41cd-a91d-05d4647b74e5-istiod-ca-cert" pod "bebbe758-a515-41cd-a91d-05d4647b74e5" (UID: "bebbe758-a515-41cd-a91d-05d4647b74e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:52.725214 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.725164 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-data\") pod \"bebbe758-a515-41cd-a91d-05d4647b74e5\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " Oct 30 19:13:52.725214 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.725199 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-credential-socket\") pod \"bebbe758-a515-41cd-a91d-05d4647b74e5\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " Oct 30 19:13:52.725430 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.725232 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-token\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-token\") pod \"bebbe758-a515-41cd-a91d-05d4647b74e5\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " Oct 30 19:13:52.725784 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.725672 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-data" pod "bebbe758-a515-41cd-a91d-05d4647b74e5" (UID: "bebbe758-a515-41cd-a91d-05d4647b74e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:52.725784 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.725683 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-credential-socket" pod "bebbe758-a515-41cd-a91d-05d4647b74e5" (UID: "bebbe758-a515-41cd-a91d-05d4647b74e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:52.725784 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.725727 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.725784 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.725749 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bebbe758-a515-41cd-a91d-05d4647b74e5-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.727396 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.727359 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-podinfo" pod "bebbe758-a515-41cd-a91d-05d4647b74e5" (UID: "bebbe758-a515-41cd-a91d-05d4647b74e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:13:52.727982 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.727842 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-kube-api-access-r6gn5" pod "bebbe758-a515-41cd-a91d-05d4647b74e5" (UID: "bebbe758-a515-41cd-a91d-05d4647b74e5"). InnerVolumeSpecName "kube-api-access-r6gn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:52.728078 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.727983 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-envoy" pod "bebbe758-a515-41cd-a91d-05d4647b74e5" (UID: "bebbe758-a515-41cd-a91d-05d4647b74e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:52.728394 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.728377 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-token" pod "bebbe758-a515-41cd-a91d-05d4647b74e5" (UID: "bebbe758-a515-41cd-a91d-05d4647b74e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:52.730607 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.730584 4862 generic.go:356] "Generic (PLEG): container finished" podID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerID="5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c" exitCode=0 Oct 30 19:13:52.730729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.730638 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" event={"ID":"bebbe758-a515-41cd-a91d-05d4647b74e5","Type":"ContainerDied","Data":"5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c"} Oct 30 19:13:52.730729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.730696 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" event={"ID":"bebbe758-a515-41cd-a91d-05d4647b74e5","Type":"ContainerDied","Data":"93eb4fc01441a71bc7a9bab11878364125b96e8b31cfd11cadf7a147d2403ef4"} Oct 30 19:13:52.730729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.730710 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm" Oct 30 19:13:52.730729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.730722 4862 scope.go:122] "RemoveContainer" containerID="5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c" Oct 30 19:13:52.752152 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.752129 4862 scope.go:122] "RemoveContainer" containerID="15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc" Oct 30 19:13:52.761133 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.761113 4862 scope.go:122] "RemoveContainer" containerID="5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c" Oct 30 19:13:52.761410 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:13:52.761390 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c\": container with ID starting with 5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c not found: ID does not exist" containerID="5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c" Oct 30 19:13:52.761489 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.761419 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c"} err="failed to get container status \"5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c\": rpc error: code = NotFound desc = could not find container \"5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c\": container with ID starting with 5b1d51ec203a45eeb4dfc428856232c7c3c62c4cc925ee3ba612d6c2ab07309c not found: ID does not exist" Oct 30 19:13:52.761489 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.761437 4862 scope.go:122] "RemoveContainer" containerID="15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc" Oct 30 19:13:52.761651 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:13:52.761637 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc\": container with ID starting with 15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc not found: ID does not exist" containerID="15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc" Oct 30 19:13:52.761694 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.761656 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc"} err="failed to get container status \"15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc\": rpc error: code = NotFound desc = could not find container \"15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc\": container with ID starting with 15b2c77912d1763466851acf4c04097cd79efcad8fe7399a354e5bd0b59580dc not found: ID does not exist" Oct 30 19:13:52.826173 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.826155 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-certs\") pod \"bebbe758-a515-41cd-a91d-05d4647b74e5\" (UID: \"bebbe758-a515-41cd-a91d-05d4647b74e5\") " Oct 30 19:13:52.826422 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.826402 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-certs" pod "bebbe758-a515-41cd-a91d-05d4647b74e5" (UID: "bebbe758-a515-41cd-a91d-05d4647b74e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:13:52.826513 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.826499 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.826557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.826514 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6gn5\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-kube-api-access-r6gn5\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.826557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.826524 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.826557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.826533 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.826557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.826541 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.826557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.826549 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bebbe758-a515-41cd-a91d-05d4647b74e5-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:52.826557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:52.826558 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bebbe758-a515-41cd-a91d-05d4647b74e5-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:13:53.065381 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:53.065346 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e4bee85-2a79-4470-a777-dc2ebb574684" path="/var/lib/kubelet/pods/0e4bee85-2a79-4470-a777-dc2ebb574684/volumes" Oct 30 19:13:53.077739 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:53.077703 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm"] Oct 30 19:13:53.085233 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:53.085208 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8b8e54-openshift-default-7c4ff64b85-kfncm"] Oct 30 19:13:55.062069 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:55.062036 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" path="/var/lib/kubelet/pods/bebbe758-a515-41cd-a91d-05d4647b74e5/volumes" Oct 30 19:13:56.507181 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:56.507144 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" podUID="b33431f9-a4e7-4691-9fbb-1226b6483fab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:13:57.731919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.731887 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx"] Oct 30 19:13:57.732405 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732380 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" Oct 30 19:13:57.732405 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732407 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732422 4862 cpu_manager.go:414] "containerMap: removing container" podUID="a6df5a42-1fc7-4159-b61a-ab0c969e5985" containerName="payload-pre-processing" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732427 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6df5a42-1fc7-4159-b61a-ab0c969e5985" containerName="payload-pre-processing" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732439 4862 cpu_manager.go:414] "containerMap: removing container" podUID="028d1f6f-f6e8-43bc-a753-ded273901271" containerName="revoke-keys" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732444 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028d1f6f-f6e8-43bc-a753-ded273901271" containerName="revoke-keys" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732452 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8345433c-4916-4e21-af2a-9a5214ed504d" containerName="payload-pre-processing" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732458 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8345433c-4916-4e21-af2a-9a5214ed504d" containerName="payload-pre-processing" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732486 4862 cpu_manager.go:414] "containerMap: removing container" podUID="0e4bee85-2a79-4470-a777-dc2ebb574684" containerName="payload-processing" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732490 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e4bee85-2a79-4470-a777-dc2ebb574684" containerName="payload-processing" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732498 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f8dc3f79-83b1-4e35-92f4-6a13a39834e3" containerName="maas-api" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732503 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8dc3f79-83b1-4e35-92f4-6a13a39834e3" containerName="maas-api" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732510 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" containerName="main" Oct 30 19:13:57.732520 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732514 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" containerName="main" Oct 30 19:13:57.732927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732590 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" Oct 30 19:13:57.732927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732600 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6df5a42-1fc7-4159-b61a-ab0c969e5985" containerName="payload-pre-processing" Oct 30 19:13:57.732927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732605 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="028d1f6f-f6e8-43bc-a753-ded273901271" containerName="revoke-keys" Oct 30 19:13:57.732927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732613 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8345433c-4916-4e21-af2a-9a5214ed504d" containerName="payload-pre-processing" Oct 30 19:13:57.732927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732623 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e4bee85-2a79-4470-a777-dc2ebb574684" containerName="payload-processing" Oct 30 19:13:57.732927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732629 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8dc3f79-83b1-4e35-92f4-6a13a39834e3" containerName="maas-api" Oct 30 19:13:57.732927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.732633 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef7feae8-dfc9-4e75-b95e-e1a29a5f1729" containerName="main" Oct 30 19:13:57.737333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.737315 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:57.750842 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.750821 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:13:57.761489 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.761464 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:13:57.765567 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.765538 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx"] Oct 30 19:13:57.780085 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.780057 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:57.780230 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.780119 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvkkp\" (UniqueName: \"kubernetes.io/projected/741da32c-b173-4e0d-966c-0ace9e06ff7a-kube-api-access-hvkkp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:57.780230 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.780216 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:57.881458 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.881427 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:57.881605 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.881505 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkkp\" (UniqueName: \"kubernetes.io/projected/741da32c-b173-4e0d-966c-0ace9e06ff7a-kube-api-access-hvkkp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:57.881605 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.881537 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:57.882187 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.882166 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:57.884014 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.883994 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:57.899657 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:57.899625 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkkp\" (UniqueName: \"kubernetes.io/projected/741da32c-b173-4e0d-966c-0ace9e06ff7a-kube-api-access-hvkkp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:58.051675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:58.051646 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:13:58.350643 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:58.350619 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx"] Oct 30 19:13:58.353589 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:13:58.353561 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741da32c_b173_4e0d_966c_0ace9e06ff7a.slice/crio-c042ed0a5a193b6b776ede04e1c2c61468033309b1f930fa98d9c250a8db1a7a WatchSource:0}: Error finding container c042ed0a5a193b6b776ede04e1c2c61468033309b1f930fa98d9c250a8db1a7a: Status 404 returned error can't find the container with id c042ed0a5a193b6b776ede04e1c2c61468033309b1f930fa98d9c250a8db1a7a Oct 30 19:13:58.779983 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:58.779945 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" event={"ID":"741da32c-b173-4e0d-966c-0ace9e06ff7a","Type":"ContainerStarted","Data":"2cb060406e85d8274b36af9fac4791d19866589605090828265f63c8b6634d25"} Oct 30 19:13:58.780376 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:58.780035 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" event={"ID":"741da32c-b173-4e0d-966c-0ace9e06ff7a","Type":"ContainerStarted","Data":"c042ed0a5a193b6b776ede04e1c2c61468033309b1f930fa98d9c250a8db1a7a"} Oct 30 19:13:58.820569 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:58.820522 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" podStartSLOduration=1.8205097430000001 podStartE2EDuration="1.820509743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:58.819756677 +0000 UTC m=+4537.868801590" watchObservedRunningTime="2026-10-30 19:13:58.820509743 +0000 UTC m=+4537.869554847" Oct 30 19:13:59.788428 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:59.788396 4862 generic.go:356] "Generic (PLEG): container finished" podID="741da32c-b173-4e0d-966c-0ace9e06ff7a" containerID="2cb060406e85d8274b36af9fac4791d19866589605090828265f63c8b6634d25" exitCode=0 Oct 30 19:13:59.788821 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:13:59.788478 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" event={"ID":"741da32c-b173-4e0d-966c-0ace9e06ff7a","Type":"ContainerDied","Data":"2cb060406e85d8274b36af9fac4791d19866589605090828265f63c8b6634d25"} Oct 30 19:14:00.936931 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:00.936906 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:14:01.042870 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.042830 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-openshift-ca\") pod \"741da32c-b173-4e0d-966c-0ace9e06ff7a\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " Oct 30 19:14:01.043043 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.042932 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741da32c-b173-4e0d-966c-0ace9e06ff7a-kube-api-access-hvkkp\" (UniqueName: \"kubernetes.io/projected/741da32c-b173-4e0d-966c-0ace9e06ff7a-kube-api-access-hvkkp\") pod \"741da32c-b173-4e0d-966c-0ace9e06ff7a\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " Oct 30 19:14:01.043139 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.043117 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-xks-ca\") pod \"741da32c-b173-4e0d-966c-0ace9e06ff7a\" (UID: \"741da32c-b173-4e0d-966c-0ace9e06ff7a\") " Oct 30 19:14:01.043289 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.043238 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-openshift-ca" pod "741da32c-b173-4e0d-966c-0ace9e06ff7a" (UID: "741da32c-b173-4e0d-966c-0ace9e06ff7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:01.043638 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.043607 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:01.045949 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.045919 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-xks-ca" pod "741da32c-b173-4e0d-966c-0ace9e06ff7a" (UID: "741da32c-b173-4e0d-966c-0ace9e06ff7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:01.048405 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.048386 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741da32c-b173-4e0d-966c-0ace9e06ff7a-kube-api-access-hvkkp" pod "741da32c-b173-4e0d-966c-0ace9e06ff7a" (UID: "741da32c-b173-4e0d-966c-0ace9e06ff7a"). InnerVolumeSpecName "kube-api-access-hvkkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:01.145186 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.145144 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvkkp\" (UniqueName: \"kubernetes.io/projected/741da32c-b173-4e0d-966c-0ace9e06ff7a-kube-api-access-hvkkp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:01.145186 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.145178 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741da32c-b173-4e0d-966c-0ace9e06ff7a-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:01.801776 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.801750 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" Oct 30 19:14:01.801958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.801786 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx" event={"ID":"741da32c-b173-4e0d-966c-0ace9e06ff7a","Type":"ContainerDied","Data":"c042ed0a5a193b6b776ede04e1c2c61468033309b1f930fa98d9c250a8db1a7a"} Oct 30 19:14:01.801958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:01.801834 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c042ed0a5a193b6b776ede04e1c2c61468033309b1f930fa98d9c250a8db1a7a" Oct 30 19:14:03.539112 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.539088 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 19:14:03.671400 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.671326 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33431f9-a4e7-4691-9fbb-1226b6483fab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33431f9-a4e7-4691-9fbb-1226b6483fab-plugins-config-volume\") pod \"b33431f9-a4e7-4691-9fbb-1226b6483fab\" (UID: \"b33431f9-a4e7-4691-9fbb-1226b6483fab\") " Oct 30 19:14:03.671400 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.671371 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33431f9-a4e7-4691-9fbb-1226b6483fab-kube-api-access-dh5vx\" (UniqueName: \"kubernetes.io/projected/b33431f9-a4e7-4691-9fbb-1226b6483fab-kube-api-access-dh5vx\") pod \"b33431f9-a4e7-4691-9fbb-1226b6483fab\" (UID: \"b33431f9-a4e7-4691-9fbb-1226b6483fab\") " Oct 30 19:14:03.671685 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.671662 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33431f9-a4e7-4691-9fbb-1226b6483fab-plugins-config-volume" pod "b33431f9-a4e7-4691-9fbb-1226b6483fab" (UID: "b33431f9-a4e7-4691-9fbb-1226b6483fab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:03.671852 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.671839 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33431f9-a4e7-4691-9fbb-1226b6483fab-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:03.673718 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.673677 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33431f9-a4e7-4691-9fbb-1226b6483fab-kube-api-access-dh5vx" pod "b33431f9-a4e7-4691-9fbb-1226b6483fab" (UID: "b33431f9-a4e7-4691-9fbb-1226b6483fab"). InnerVolumeSpecName "kube-api-access-dh5vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:03.773577 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.773537 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh5vx\" (UniqueName: \"kubernetes.io/projected/b33431f9-a4e7-4691-9fbb-1226b6483fab-kube-api-access-dh5vx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:03.814918 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.814889 4862 generic.go:356] "Generic (PLEG): container finished" podID="b33431f9-a4e7-4691-9fbb-1226b6483fab" containerID="9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59" exitCode=0 Oct 30 19:14:03.815028 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.814932 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" Oct 30 19:14:03.815028 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.814973 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" event={"ID":"b33431f9-a4e7-4691-9fbb-1226b6483fab","Type":"ContainerDied","Data":"9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59"} Oct 30 19:14:03.815116 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.815038 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5" event={"ID":"b33431f9-a4e7-4691-9fbb-1226b6483fab","Type":"ContainerDied","Data":"f13bb482217b5f1a069c07aa02efa98f74442cd6ceb228d2f3945b55cc903942"} Oct 30 19:14:03.815116 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.815061 4862 scope.go:122] "RemoveContainer" containerID="9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59" Oct 30 19:14:03.829431 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.829405 4862 scope.go:122] "RemoveContainer" containerID="9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59" Oct 30 19:14:03.831169 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:14:03.831133 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59\": container with ID starting with 9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59 not found: ID does not exist" containerID="9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59" Oct 30 19:14:03.831360 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.831177 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59"} err="failed to get container status \"9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59\": rpc error: code = NotFound desc = could not find container \"9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59\": container with ID starting with 9ec4d7efd1f39b4ba615aaa9343c1ec0a7de54d20b1c3259bc49ee0634f75c59 not found: ID does not exist" Oct 30 19:14:03.864152 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.864119 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5"] Oct 30 19:14:03.868749 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:03.868714 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6babca-6f69b7f9-tlrg5"] Oct 30 19:14:05.060763 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:05.060719 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33431f9-a4e7-4691-9fbb-1226b6483fab" path="/var/lib/kubelet/pods/b33431f9-a4e7-4691-9fbb-1226b6483fab/volumes" Oct 30 19:14:09.479961 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:09.479925 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn"] Oct 30 19:14:09.480645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:09.480228 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" containerID="cri-o://c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2" gracePeriod=30 Oct 30 19:14:10.078134 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.078099 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs"] Oct 30 19:14:10.078802 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.078779 4862 cpu_manager.go:414] "containerMap: removing container" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" Oct 30 19:14:10.078924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.078803 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" Oct 30 19:14:10.078924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.078851 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b33431f9-a4e7-4691-9fbb-1226b6483fab" containerName="payload-pre-processing" Oct 30 19:14:10.078924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.078861 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33431f9-a4e7-4691-9fbb-1226b6483fab" containerName="payload-pre-processing" Oct 30 19:14:10.078924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.078882 4862 cpu_manager.go:414] "containerMap: removing container" podUID="741da32c-b173-4e0d-966c-0ace9e06ff7a" containerName="revoke-keys" Oct 30 19:14:10.078924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.078890 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741da32c-b173-4e0d-966c-0ace9e06ff7a" containerName="revoke-keys" Oct 30 19:14:10.079198 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.079014 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33431f9-a4e7-4691-9fbb-1226b6483fab" containerName="payload-pre-processing" Oct 30 19:14:10.079198 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.079030 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="741da32c-b173-4e0d-966c-0ace9e06ff7a" containerName="revoke-keys" Oct 30 19:14:10.079198 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.079045 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="bebbe758-a515-41cd-a91d-05d4647b74e5" containerName="istio-proxy" Oct 30 19:14:10.083747 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.083726 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.097193 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.097168 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:14:10.097340 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.097317 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:14:10.100991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.100966 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs"] Oct 30 19:14:10.244651 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.244617 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.244839 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.244684 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.244839 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.244826 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgpnf\" (UniqueName: \"kubernetes.io/projected/b7206440-e5b0-4ac5-893b-9fd3f1d46994-kube-api-access-wgpnf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.345966 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.345897 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.345966 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.345954 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.346132 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.346007 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgpnf\" (UniqueName: \"kubernetes.io/projected/b7206440-e5b0-4ac5-893b-9fd3f1d46994-kube-api-access-wgpnf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.346574 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.346550 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.348497 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.348471 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.363511 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.363486 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgpnf\" (UniqueName: \"kubernetes.io/projected/b7206440-e5b0-4ac5-893b-9fd3f1d46994-kube-api-access-wgpnf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.397480 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.397462 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:10.696848 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.696786 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs"] Oct 30 19:14:10.699205 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:14:10.699175 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7206440_e5b0_4ac5_893b_9fd3f1d46994.slice/crio-5d18990d063b41b08799071a31d71fb0414b6d00b6143156991ec5bd873cfc0d WatchSource:0}: Error finding container 5d18990d063b41b08799071a31d71fb0414b6d00b6143156991ec5bd873cfc0d: Status 404 returned error can't find the container with id 5d18990d063b41b08799071a31d71fb0414b6d00b6143156991ec5bd873cfc0d Oct 30 19:14:10.859013 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:10.858977 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" event={"ID":"b7206440-e5b0-4ac5-893b-9fd3f1d46994","Type":"ContainerStarted","Data":"5d18990d063b41b08799071a31d71fb0414b6d00b6143156991ec5bd873cfc0d"} Oct 30 19:14:11.867525 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:11.867491 4862 generic.go:356] "Generic (PLEG): container finished" podID="b7206440-e5b0-4ac5-893b-9fd3f1d46994" containerID="04d9365b85544014711a8c545fcd58170214c32538ecd4839cfa430734b29079" exitCode=0 Oct 30 19:14:11.867944 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:11.867581 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" event={"ID":"b7206440-e5b0-4ac5-893b-9fd3f1d46994","Type":"ContainerDied","Data":"04d9365b85544014711a8c545fcd58170214c32538ecd4839cfa430734b29079"} Oct 30 19:14:13.019568 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.019544 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:13.177618 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.177542 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7206440-e5b0-4ac5-893b-9fd3f1d46994-kube-api-access-wgpnf\" (UniqueName: \"kubernetes.io/projected/b7206440-e5b0-4ac5-893b-9fd3f1d46994-kube-api-access-wgpnf\") pod \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " Oct 30 19:14:13.177763 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.177746 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-xks-ca\") pod \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " Oct 30 19:14:13.177810 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.177795 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-openshift-ca\") pod \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\" (UID: \"b7206440-e5b0-4ac5-893b-9fd3f1d46994\") " Oct 30 19:14:13.178192 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.178164 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-openshift-ca" pod "b7206440-e5b0-4ac5-893b-9fd3f1d46994" (UID: "b7206440-e5b0-4ac5-893b-9fd3f1d46994"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:13.179943 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.179919 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7206440-e5b0-4ac5-893b-9fd3f1d46994-kube-api-access-wgpnf" pod "b7206440-e5b0-4ac5-893b-9fd3f1d46994" (UID: "b7206440-e5b0-4ac5-893b-9fd3f1d46994"). InnerVolumeSpecName "kube-api-access-wgpnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:13.180064 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.180012 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-xks-ca" pod "b7206440-e5b0-4ac5-893b-9fd3f1d46994" (UID: "b7206440-e5b0-4ac5-893b-9fd3f1d46994"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:13.278754 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.278723 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgpnf\" (UniqueName: \"kubernetes.io/projected/b7206440-e5b0-4ac5-893b-9fd3f1d46994-kube-api-access-wgpnf\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:13.278754 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.278750 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:13.278921 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.278764 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7206440-e5b0-4ac5-893b-9fd3f1d46994-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:13.882442 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.882407 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" Oct 30 19:14:13.882622 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.882462 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs" event={"ID":"b7206440-e5b0-4ac5-893b-9fd3f1d46994","Type":"ContainerDied","Data":"5d18990d063b41b08799071a31d71fb0414b6d00b6143156991ec5bd873cfc0d"} Oct 30 19:14:13.882622 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:13.882516 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d18990d063b41b08799071a31d71fb0414b6d00b6143156991ec5bd873cfc0d" Oct 30 19:14:14.444029 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.443999 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:14:14.752983 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.752960 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 19:14:14.796187 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796160 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-kube-api-access-bw5d9\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-kube-api-access-bw5d9\") pod \"757179a3-982f-402c-8b8a-00acf2cc8f56\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " Oct 30 19:14:14.796374 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796204 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-certs\") pod \"757179a3-982f-402c-8b8a-00acf2cc8f56\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " Oct 30 19:14:14.796374 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796307 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-credential-socket\") pod \"757179a3-982f-402c-8b8a-00acf2cc8f56\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " Oct 30 19:14:14.796504 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796379 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-podinfo\") pod \"757179a3-982f-402c-8b8a-00acf2cc8f56\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " Oct 30 19:14:14.796504 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796408 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-token\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-token\") pod \"757179a3-982f-402c-8b8a-00acf2cc8f56\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " Oct 30 19:14:14.796504 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796455 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-data\") pod \"757179a3-982f-402c-8b8a-00acf2cc8f56\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " Oct 30 19:14:14.797092 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796509 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-envoy\") pod \"757179a3-982f-402c-8b8a-00acf2cc8f56\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " Oct 30 19:14:14.797092 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796525 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-certs" pod "757179a3-982f-402c-8b8a-00acf2cc8f56" (UID: "757179a3-982f-402c-8b8a-00acf2cc8f56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.797092 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796570 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-socket\") pod \"757179a3-982f-402c-8b8a-00acf2cc8f56\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " Oct 30 19:14:14.797092 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.796667 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757179a3-982f-402c-8b8a-00acf2cc8f56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/757179a3-982f-402c-8b8a-00acf2cc8f56-istiod-ca-cert\") pod \"757179a3-982f-402c-8b8a-00acf2cc8f56\" (UID: \"757179a3-982f-402c-8b8a-00acf2cc8f56\") " Oct 30 19:14:14.797356 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.797230 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.797585 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.797561 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757179a3-982f-402c-8b8a-00acf2cc8f56-istiod-ca-cert" pod "757179a3-982f-402c-8b8a-00acf2cc8f56" (UID: "757179a3-982f-402c-8b8a-00acf2cc8f56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:14.797702 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.797650 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-data" pod "757179a3-982f-402c-8b8a-00acf2cc8f56" (UID: "757179a3-982f-402c-8b8a-00acf2cc8f56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.797762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.797720 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-credential-socket" pod "757179a3-982f-402c-8b8a-00acf2cc8f56" (UID: "757179a3-982f-402c-8b8a-00acf2cc8f56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.797762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.797729 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-socket" pod "757179a3-982f-402c-8b8a-00acf2cc8f56" (UID: "757179a3-982f-402c-8b8a-00acf2cc8f56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.799394 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.799367 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-kube-api-access-bw5d9" pod "757179a3-982f-402c-8b8a-00acf2cc8f56" (UID: "757179a3-982f-402c-8b8a-00acf2cc8f56"). InnerVolumeSpecName "kube-api-access-bw5d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:14.799844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.799824 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-podinfo" pod "757179a3-982f-402c-8b8a-00acf2cc8f56" (UID: "757179a3-982f-402c-8b8a-00acf2cc8f56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:14.799844 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.799834 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-token" pod "757179a3-982f-402c-8b8a-00acf2cc8f56" (UID: "757179a3-982f-402c-8b8a-00acf2cc8f56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:14.799970 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.799886 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-envoy" pod "757179a3-982f-402c-8b8a-00acf2cc8f56" (UID: "757179a3-982f-402c-8b8a-00acf2cc8f56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:14.891496 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.891465 4862 generic.go:356] "Generic (PLEG): container finished" podID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerID="c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2" exitCode=0 Oct 30 19:14:14.891666 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.891523 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" event={"ID":"757179a3-982f-402c-8b8a-00acf2cc8f56","Type":"ContainerDied","Data":"c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2"} Oct 30 19:14:14.891666 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.891541 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" Oct 30 19:14:14.891666 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.891578 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn" event={"ID":"757179a3-982f-402c-8b8a-00acf2cc8f56","Type":"ContainerDied","Data":"b15cdd6c67ebe9f180ba8591bbf9c7d999320f8b7c9f4666dcad57c313433efc"} Oct 30 19:14:14.891666 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.891601 4862 scope.go:122] "RemoveContainer" containerID="c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2" Oct 30 19:14:14.900832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.899054 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.900832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.899081 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.900832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.899096 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.900832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.899116 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.900832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.899130 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.900832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.899145 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/757179a3-982f-402c-8b8a-00acf2cc8f56-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.900832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.899160 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw5d9\" (UniqueName: \"kubernetes.io/projected/757179a3-982f-402c-8b8a-00acf2cc8f56-kube-api-access-bw5d9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.900832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.899178 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/757179a3-982f-402c-8b8a-00acf2cc8f56-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:14.907466 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.907441 4862 scope.go:122] "RemoveContainer" containerID="e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894" Oct 30 19:14:14.919644 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.919621 4862 scope.go:122] "RemoveContainer" containerID="c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2" Oct 30 19:14:14.919979 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:14:14.919955 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2\": container with ID starting with c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2 not found: ID does not exist" containerID="c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2" Oct 30 19:14:14.920040 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.919988 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2"} err="failed to get container status \"c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2\": rpc error: code = NotFound desc = could not find container \"c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2\": container with ID starting with c162c853add5a6185e8f8e08e95152420ea3ffd1f9cecddb1261fd90f52061c2 not found: ID does not exist" Oct 30 19:14:14.920040 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.920005 4862 scope.go:122] "RemoveContainer" containerID="e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894" Oct 30 19:14:14.920292 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:14:14.920241 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894\": container with ID starting with e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894 not found: ID does not exist" containerID="e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894" Oct 30 19:14:14.920349 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.920295 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894"} err="failed to get container status \"e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894\": rpc error: code = NotFound desc = could not find container \"e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894\": container with ID starting with e65612fd17a86d78bd1e653d19c8817792128cc06722a5304e506a39ab8b2894 not found: ID does not exist" Oct 30 19:14:14.940435 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.940411 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn"] Oct 30 19:14:14.945747 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:14.945724 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-39c99d-openshift-default-5c9b64c6d5-4d6mn"] Oct 30 19:14:15.061677 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:15.061652 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" path="/var/lib/kubelet/pods/757179a3-982f-402c-8b8a-00acf2cc8f56/volumes" Oct 30 19:14:16.729627 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:16.729594 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx"] Oct 30 19:14:16.740723 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:16.740692 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1a2436-hkccx"] Oct 30 19:14:17.061324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:17.061292 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741da32c-b173-4e0d-966c-0ace9e06ff7a" path="/var/lib/kubelet/pods/741da32c-b173-4e0d-966c-0ace9e06ff7a/volumes" Oct 30 19:14:19.870538 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:14:19.870503 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:14:24.098040 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.098005 4862 scope.go:122] "RemoveContainer" containerID="4e02106610137241a7790ce973f0878536c1d5999916a13925363f590f81ced6" Oct 30 19:14:24.110513 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.110486 4862 scope.go:122] "RemoveContainer" containerID="f9ee77d66dc2259a149153debd4f32d7bd425f8ddeb8adc26d69c436684c0358" Oct 30 19:14:24.120405 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.120382 4862 scope.go:122] "RemoveContainer" containerID="d42ce3a2c0c50f32144cc930e7740fb9e76374b8591b7895dad704e10fae253c" Oct 30 19:14:24.831504 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.831469 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr"] Oct 30 19:14:24.831934 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.831921 4862 cpu_manager.go:414] "containerMap: removing container" podUID="b7206440-e5b0-4ac5-893b-9fd3f1d46994" containerName="revoke-keys" Oct 30 19:14:24.831980 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.831936 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7206440-e5b0-4ac5-893b-9fd3f1d46994" containerName="revoke-keys" Oct 30 19:14:24.831980 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.831959 4862 cpu_manager.go:414] "containerMap: removing container" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" Oct 30 19:14:24.831980 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.831964 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" Oct 30 19:14:24.832078 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.832052 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7206440-e5b0-4ac5-893b-9fd3f1d46994" containerName="revoke-keys" Oct 30 19:14:24.832078 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.832063 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" Oct 30 19:14:24.832078 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.832072 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" Oct 30 19:14:24.837395 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.837373 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:24.895199 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.895167 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:14:24.895437 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.895316 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:14:24.899133 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.899105 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr"] Oct 30 19:14:24.906337 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.906318 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxjsm\" (UniqueName: \"kubernetes.io/projected/c1b784de-ccfc-4437-a107-ac8e35916058-kube-api-access-jxjsm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:24.906454 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.906378 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:24.906454 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:24.906434 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:25.007463 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.007440 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxjsm\" (UniqueName: \"kubernetes.io/projected/c1b784de-ccfc-4437-a107-ac8e35916058-kube-api-access-jxjsm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:25.007593 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.007494 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:25.007593 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.007514 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:25.008130 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.008107 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:25.009890 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.009864 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:25.025037 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.025015 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxjsm\" (UniqueName: \"kubernetes.io/projected/c1b784de-ccfc-4437-a107-ac8e35916058-kube-api-access-jxjsm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:25.151046 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.150993 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:25.462425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.462395 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr"] Oct 30 19:14:25.464431 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:14:25.464401 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b784de_ccfc_4437_a107_ac8e35916058.slice/crio-e84b1d530fad8e5ae9e8871e4ef1d3f23861c7781c7aca125cc2bc1c19419e2c WatchSource:0}: Error finding container e84b1d530fad8e5ae9e8871e4ef1d3f23861c7781c7aca125cc2bc1c19419e2c: Status 404 returned error can't find the container with id e84b1d530fad8e5ae9e8871e4ef1d3f23861c7781c7aca125cc2bc1c19419e2c Oct 30 19:14:25.962018 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.961979 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" event={"ID":"c1b784de-ccfc-4437-a107-ac8e35916058","Type":"ContainerStarted","Data":"0e0821864add9106cfa6a80ec09cb1fdb37ec2241ffa49038d9573ca9e6b66a1"} Oct 30 19:14:25.962167 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:25.962046 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" event={"ID":"c1b784de-ccfc-4437-a107-ac8e35916058","Type":"ContainerStarted","Data":"e84b1d530fad8e5ae9e8871e4ef1d3f23861c7781c7aca125cc2bc1c19419e2c"} Oct 30 19:14:26.010962 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:26.010853 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" podStartSLOduration=2.010833359 podStartE2EDuration="2.010833359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:25.996518692 +0000 UTC m=+4565.045563606" watchObservedRunningTime="2026-10-30 19:14:26.010833359 +0000 UTC m=+4565.059878275" Oct 30 19:14:26.969327 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:26.969255 4862 generic.go:356] "Generic (PLEG): container finished" podID="c1b784de-ccfc-4437-a107-ac8e35916058" containerID="0e0821864add9106cfa6a80ec09cb1fdb37ec2241ffa49038d9573ca9e6b66a1" exitCode=0 Oct 30 19:14:26.969736 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:26.969403 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" event={"ID":"c1b784de-ccfc-4437-a107-ac8e35916058","Type":"ContainerDied","Data":"0e0821864add9106cfa6a80ec09cb1fdb37ec2241ffa49038d9573ca9e6b66a1"} Oct 30 19:14:28.122940 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.122915 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:28.169359 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.169335 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-xks-ca\") pod \"c1b784de-ccfc-4437-a107-ac8e35916058\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " Oct 30 19:14:28.169528 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.169510 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-openshift-ca\") pod \"c1b784de-ccfc-4437-a107-ac8e35916058\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " Oct 30 19:14:28.169617 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.169585 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b784de-ccfc-4437-a107-ac8e35916058-kube-api-access-jxjsm\" (UniqueName: \"kubernetes.io/projected/c1b784de-ccfc-4437-a107-ac8e35916058-kube-api-access-jxjsm\") pod \"c1b784de-ccfc-4437-a107-ac8e35916058\" (UID: \"c1b784de-ccfc-4437-a107-ac8e35916058\") " Oct 30 19:14:28.169857 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.169836 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-openshift-ca" pod "c1b784de-ccfc-4437-a107-ac8e35916058" (UID: "c1b784de-ccfc-4437-a107-ac8e35916058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:28.170121 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.170103 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:28.171613 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.171590 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-xks-ca" pod "c1b784de-ccfc-4437-a107-ac8e35916058" (UID: "c1b784de-ccfc-4437-a107-ac8e35916058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:14:28.171939 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.171922 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b784de-ccfc-4437-a107-ac8e35916058-kube-api-access-jxjsm" pod "c1b784de-ccfc-4437-a107-ac8e35916058" (UID: "c1b784de-ccfc-4437-a107-ac8e35916058"). InnerVolumeSpecName "kube-api-access-jxjsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:28.270928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.270858 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxjsm\" (UniqueName: \"kubernetes.io/projected/c1b784de-ccfc-4437-a107-ac8e35916058-kube-api-access-jxjsm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:28.270928 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.270887 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b784de-ccfc-4437-a107-ac8e35916058-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:28.982945 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.982915 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" Oct 30 19:14:28.983162 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.982947 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr" event={"ID":"c1b784de-ccfc-4437-a107-ac8e35916058","Type":"ContainerDied","Data":"e84b1d530fad8e5ae9e8871e4ef1d3f23861c7781c7aca125cc2bc1c19419e2c"} Oct 30 19:14:28.983162 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:28.982992 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e84b1d530fad8e5ae9e8871e4ef1d3f23861c7781c7aca125cc2bc1c19419e2c" Oct 30 19:14:32.369245 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:32.369209 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs"] Oct 30 19:14:32.379128 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:32.379102 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-236fc4-8pcvs"] Oct 30 19:14:33.061641 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:33.061608 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7206440-e5b0-4ac5-893b-9fd3f1d46994" path="/var/lib/kubelet/pods/b7206440-e5b0-4ac5-893b-9fd3f1d46994/volumes" Oct 30 19:14:35.463372 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:35.463336 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88"] Oct 30 19:14:36.066291 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:36.063475 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg"] Oct 30 19:14:36.079896 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:36.067627 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a205a9-cdvmg"] Oct 30 19:14:37.061058 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:37.061028 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9e330dc-6b6b-4a18-ab80-1979b576b17a" path="/var/lib/kubelet/pods/f9e330dc-6b6b-4a18-ab80-1979b576b17a/volumes" Oct 30 19:14:37.534166 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:37.534089 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:14:40.464547 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:40.464499 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" podUID="56f556fe-3216-4078-af8d-920b06f41a7b" containerName="payload-pre-processing" containerID="cri-o://cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030" gracePeriod=25 Oct 30 19:14:42.505150 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:42.505117 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" podUID="56f556fe-3216-4078-af8d-920b06f41a7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:44.971332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:44.971295 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf"] Oct 30 19:14:45.005634 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:45.005604 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw"] Oct 30 19:14:47.041371 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:47.041247 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr"] Oct 30 19:14:47.051293 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:47.051257 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-24b124-kqhdr"] Oct 30 19:14:47.061322 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:47.061294 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b784de-ccfc-4437-a107-ac8e35916058" path="/var/lib/kubelet/pods/c1b784de-ccfc-4437-a107-ac8e35916058/volumes" Oct 30 19:14:49.869614 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:14:49.869574 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:14:49.971924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:49.971870 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" podUID="d3bb768e-e677-44c7-a203-b02125cb3384" containerName="payload-processing" containerID="cri-o://97e686d078905234a44d7d5d989d9e825b2520e102d2513f6f8a5f83e4d82310" gracePeriod=25 Oct 30 19:14:50.006319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:50.006283 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" podUID="8d32fa3e-7c31-403d-bc9d-099927dae6e8" containerName="payload-pre-processing" containerID="cri-o://c4e0f20eb14d79eab4151d1decd58777dc8db503ff3ddcf89223fbbf5c5610e6" gracePeriod=25 Oct 30 19:14:51.109161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:51.109132 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:14:51.480830 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:51.480744 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" podUID="d3bb768e-e677-44c7-a203-b02125cb3384" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:51.480830 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:51.480761 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" podUID="8d32fa3e-7c31-403d-bc9d-099927dae6e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:52.505577 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:52.505536 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" podUID="56f556fe-3216-4078-af8d-920b06f41a7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:14:54.174068 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:54.174025 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t"] Oct 30 19:14:54.174590 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:54.174294 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerName="istio-proxy" containerID="cri-o://3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec" gracePeriod=30 Oct 30 19:14:59.439963 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.439910 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:14:59.546770 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.546740 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-kube-api-access-8v9cx\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-kube-api-access-8v9cx\") pod \"4271c0fb-4068-4a80-a095-418dad3b8c60\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " Oct 30 19:14:59.546948 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.546806 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-envoy\") pod \"4271c0fb-4068-4a80-a095-418dad3b8c60\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " Oct 30 19:14:59.546948 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.546855 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-socket\") pod \"4271c0fb-4068-4a80-a095-418dad3b8c60\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " Oct 30 19:14:59.547079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.546958 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4271c0fb-4068-4a80-a095-418dad3b8c60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4271c0fb-4068-4a80-a095-418dad3b8c60-istiod-ca-cert\") pod \"4271c0fb-4068-4a80-a095-418dad3b8c60\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " Oct 30 19:14:59.547079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.546987 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-data\") pod \"4271c0fb-4068-4a80-a095-418dad3b8c60\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " Oct 30 19:14:59.547079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547036 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-certs\") pod \"4271c0fb-4068-4a80-a095-418dad3b8c60\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " Oct 30 19:14:59.547237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547077 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-podinfo\") pod \"4271c0fb-4068-4a80-a095-418dad3b8c60\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " Oct 30 19:14:59.547237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547166 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-credential-socket\") pod \"4271c0fb-4068-4a80-a095-418dad3b8c60\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " Oct 30 19:14:59.547237 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547196 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-socket" pod "4271c0fb-4068-4a80-a095-418dad3b8c60" (UID: "4271c0fb-4068-4a80-a095-418dad3b8c60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.547441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547241 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-token\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-token\") pod \"4271c0fb-4068-4a80-a095-418dad3b8c60\" (UID: \"4271c0fb-4068-4a80-a095-418dad3b8c60\") " Oct 30 19:14:59.547441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547297 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-data" pod "4271c0fb-4068-4a80-a095-418dad3b8c60" (UID: "4271c0fb-4068-4a80-a095-418dad3b8c60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.547441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547382 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-certs" pod "4271c0fb-4068-4a80-a095-418dad3b8c60" (UID: "4271c0fb-4068-4a80-a095-418dad3b8c60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.547441 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547387 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4271c0fb-4068-4a80-a095-418dad3b8c60-istiod-ca-cert" pod "4271c0fb-4068-4a80-a095-418dad3b8c60" (UID: "4271c0fb-4068-4a80-a095-418dad3b8c60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:59.547651 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547527 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-credential-socket" pod "4271c0fb-4068-4a80-a095-418dad3b8c60" (UID: "4271c0fb-4068-4a80-a095-418dad3b8c60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.547746 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547729 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.547811 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547748 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.547811 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547758 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.547811 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547767 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4271c0fb-4068-4a80-a095-418dad3b8c60-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.547811 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.547778 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.549645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.549611 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-podinfo" pod "4271c0fb-4068-4a80-a095-418dad3b8c60" (UID: "4271c0fb-4068-4a80-a095-418dad3b8c60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:59.549645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.549635 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-token" pod "4271c0fb-4068-4a80-a095-418dad3b8c60" (UID: "4271c0fb-4068-4a80-a095-418dad3b8c60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:59.549830 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.549679 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-kube-api-access-8v9cx" pod "4271c0fb-4068-4a80-a095-418dad3b8c60" (UID: "4271c0fb-4068-4a80-a095-418dad3b8c60"). InnerVolumeSpecName "kube-api-access-8v9cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:59.549830 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.549743 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-envoy" pod "4271c0fb-4068-4a80-a095-418dad3b8c60" (UID: "4271c0fb-4068-4a80-a095-418dad3b8c60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:59.633534 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.633507 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw"] Oct 30 19:14:59.648920 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.648895 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.648920 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.648920 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.649135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.648930 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v9cx\" (UniqueName: \"kubernetes.io/projected/4271c0fb-4068-4a80-a095-418dad3b8c60-kube-api-access-8v9cx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.649135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.648940 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4271c0fb-4068-4a80-a095-418dad3b8c60-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:14:59.656566 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:14:59.656544 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw"] Oct 30 19:15:00.191447 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.191412 4862 generic.go:356] "Generic (PLEG): container finished" podID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerID="3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec" exitCode=0 Oct 30 19:15:00.191616 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.191458 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" Oct 30 19:15:00.191616 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.191488 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" event={"ID":"4271c0fb-4068-4a80-a095-418dad3b8c60","Type":"ContainerDied","Data":"3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec"} Oct 30 19:15:00.191616 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.191541 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t" event={"ID":"4271c0fb-4068-4a80-a095-418dad3b8c60","Type":"ContainerDied","Data":"70239678e70e59b69b4e365b19807d8a6af04e28f820bf999e00a6609cd41d6f"} Oct 30 19:15:00.191616 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.191564 4862 scope.go:122] "RemoveContainer" containerID="3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec" Oct 30 19:15:00.202822 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.202804 4862 scope.go:122] "RemoveContainer" containerID="3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec" Oct 30 19:15:00.203098 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:00.203079 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec\": container with ID starting with 3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec not found: ID does not exist" containerID="3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec" Oct 30 19:15:00.203147 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.203106 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec"} err="failed to get container status \"3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec\": rpc error: code = NotFound desc = could not find container \"3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec\": container with ID starting with 3e26fee05bf65caa7d58bdc204faab5b29321e6f4eafedfc63cc00275bd9beec not found: ID does not exist" Oct 30 19:15:00.270061 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.270032 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t"] Oct 30 19:15:00.286028 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.285998 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-24b124-openshift-default-77557bb57f-qlr4t"] Oct 30 19:15:00.525067 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:00.525034 4862 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f556fe_3216_4078_af8d_920b06f41a7b.slice/crio-conmon-cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f556fe_3216_4078_af8d_920b06f41a7b.slice/crio-cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:15:00.734143 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.734070 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 19:15:00.862729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.862696 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f556fe-3216-4078-af8d-920b06f41a7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f556fe-3216-4078-af8d-920b06f41a7b-plugins-config-volume\") pod \"56f556fe-3216-4078-af8d-920b06f41a7b\" (UID: \"56f556fe-3216-4078-af8d-920b06f41a7b\") " Oct 30 19:15:00.862907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.862860 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f556fe-3216-4078-af8d-920b06f41a7b-kube-api-access-sbtj9\" (UniqueName: \"kubernetes.io/projected/56f556fe-3216-4078-af8d-920b06f41a7b-kube-api-access-sbtj9\") pod \"56f556fe-3216-4078-af8d-920b06f41a7b\" (UID: \"56f556fe-3216-4078-af8d-920b06f41a7b\") " Oct 30 19:15:00.863147 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.863121 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f556fe-3216-4078-af8d-920b06f41a7b-plugins-config-volume" pod "56f556fe-3216-4078-af8d-920b06f41a7b" (UID: "56f556fe-3216-4078-af8d-920b06f41a7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:00.865321 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.865296 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f556fe-3216-4078-af8d-920b06f41a7b-kube-api-access-sbtj9" pod "56f556fe-3216-4078-af8d-920b06f41a7b" (UID: "56f556fe-3216-4078-af8d-920b06f41a7b"). InnerVolumeSpecName "kube-api-access-sbtj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:00.964785 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.964748 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f556fe-3216-4078-af8d-920b06f41a7b-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:00.964785 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:00.964780 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbtj9\" (UniqueName: \"kubernetes.io/projected/56f556fe-3216-4078-af8d-920b06f41a7b-kube-api-access-sbtj9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:01.061058 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.061015 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" path="/var/lib/kubelet/pods/4271c0fb-4068-4a80-a095-418dad3b8c60/volumes" Oct 30 19:15:01.200161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.200124 4862 generic.go:356] "Generic (PLEG): container finished" podID="56f556fe-3216-4078-af8d-920b06f41a7b" containerID="cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030" exitCode=0 Oct 30 19:15:01.200358 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.200179 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" Oct 30 19:15:01.200358 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.200190 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" event={"ID":"56f556fe-3216-4078-af8d-920b06f41a7b","Type":"ContainerDied","Data":"cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030"} Oct 30 19:15:01.200358 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.200242 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88" event={"ID":"56f556fe-3216-4078-af8d-920b06f41a7b","Type":"ContainerDied","Data":"d5fa70c8e429bb88954f15df45a27af79953115d37a09e1f0c6f0e4f66debcb1"} Oct 30 19:15:01.200358 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.200289 4862 scope.go:122] "RemoveContainer" containerID="cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030" Oct 30 19:15:01.210483 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.210463 4862 scope.go:122] "RemoveContainer" containerID="cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030" Oct 30 19:15:01.210746 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:01.210728 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030\": container with ID starting with cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030 not found: ID does not exist" containerID="cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030" Oct 30 19:15:01.210804 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.210755 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030"} err="failed to get container status \"cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030\": rpc error: code = NotFound desc = could not find container \"cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030\": container with ID starting with cc4c262531e693f1aec6bbd874e3a89f5a8f84d85ccc7f52b547282530ff3030 not found: ID does not exist" Oct 30 19:15:01.246941 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.246912 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88"] Oct 30 19:15:01.250319 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.250294 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-24b124-b67d5558c-gcv88"] Oct 30 19:15:01.480735 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.480652 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" podUID="d3bb768e-e677-44c7-a203-b02125cb3384" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:01.480894 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.480809 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" podUID="8d32fa3e-7c31-403d-bc9d-099927dae6e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:15:01.842539 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:01.842496 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:15:03.030236 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:03.030204 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725"] Oct 30 19:15:03.040401 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:03.040371 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c318003-84725"] Oct 30 19:15:03.062167 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:03.062136 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f556fe-3216-4078-af8d-920b06f41a7b" path="/var/lib/kubelet/pods/56f556fe-3216-4078-af8d-920b06f41a7b/volumes" Oct 30 19:15:03.062778 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:03.062755 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc43bf9-1f43-473c-be5b-7130e798afd3" path="/var/lib/kubelet/pods/5cc43bf9-1f43-473c-be5b-7130e798afd3/volumes" Oct 30 19:15:04.395444 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.395395 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744"] Oct 30 19:15:04.395902 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.395661 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" containerID="cri-o://e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd" gracePeriod=30 Oct 30 19:15:04.899207 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899173 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr"] Oct 30 19:15:04.899638 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899624 4862 cpu_manager.go:414] "containerMap: removing container" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerName="istio-proxy" Oct 30 19:15:04.899697 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899639 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerName="istio-proxy" Oct 30 19:15:04.899697 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899656 4862 cpu_manager.go:414] "containerMap: removing container" podUID="c1b784de-ccfc-4437-a107-ac8e35916058" containerName="revoke-keys" Oct 30 19:15:04.899697 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899661 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b784de-ccfc-4437-a107-ac8e35916058" containerName="revoke-keys" Oct 30 19:15:04.899697 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899675 4862 cpu_manager.go:414] "containerMap: removing container" podUID="56f556fe-3216-4078-af8d-920b06f41a7b" containerName="payload-pre-processing" Oct 30 19:15:04.899697 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899680 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f556fe-3216-4078-af8d-920b06f41a7b" containerName="payload-pre-processing" Oct 30 19:15:04.899697 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899695 4862 cpu_manager.go:414] "containerMap: removing container" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" Oct 30 19:15:04.899919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899700 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757179a3-982f-402c-8b8a-00acf2cc8f56" containerName="istio-proxy" Oct 30 19:15:04.899919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899779 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="4271c0fb-4068-4a80-a095-418dad3b8c60" containerName="istio-proxy" Oct 30 19:15:04.899919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899789 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b784de-ccfc-4437-a107-ac8e35916058" containerName="revoke-keys" Oct 30 19:15:04.899919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.899795 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f556fe-3216-4078-af8d-920b06f41a7b" containerName="payload-pre-processing" Oct 30 19:15:04.903719 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.903701 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:04.922023 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.921994 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr"] Oct 30 19:15:04.944776 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.944748 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:04.944914 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:04.944900 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:15:05.007883 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.007856 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.007996 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.007952 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.008104 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.008090 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pt4\" (UniqueName: \"kubernetes.io/projected/62031fd1-034c-4b62-bf54-9a2f06d66278-kube-api-access-55pt4\") pod \"maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.109011 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.108988 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pt4\" (UniqueName: \"kubernetes.io/projected/62031fd1-034c-4b62-bf54-9a2f06d66278-kube-api-access-55pt4\") pod \"maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.109145 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.109046 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.109145 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.109077 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.109715 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.109697 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.111590 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.111569 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.128175 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.128157 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pt4\" (UniqueName: \"kubernetes.io/projected/62031fd1-034c-4b62-bf54-9a2f06d66278-kube-api-access-55pt4\") pod \"maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.216434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.216368 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:05.484752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.484666 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh"] Oct 30 19:15:05.485187 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.484922 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" containerID="cri-o://8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6" gracePeriod=30 Oct 30 19:15:05.526540 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:05.526498 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr"] Oct 30 19:15:05.529836 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:15:05.529807 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62031fd1_034c_4b62_bf54_9a2f06d66278.slice/crio-c21921b143c83d0b6bdb60dc5aa27fea09728673e04becf8917febd61fbc05a6 WatchSource:0}: Error finding container c21921b143c83d0b6bdb60dc5aa27fea09728673e04becf8917febd61fbc05a6: Status 404 returned error can't find the container with id c21921b143c83d0b6bdb60dc5aa27fea09728673e04becf8917febd61fbc05a6 Oct 30 19:15:06.246050 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:06.246017 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" event={"ID":"62031fd1-034c-4b62-bf54-9a2f06d66278","Type":"ContainerStarted","Data":"031e9a9d6788ecf74f0df546c919df87dba1288fe1ee383e341ceeaa1126f84b"} Oct 30 19:15:06.246539 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:06.246072 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" event={"ID":"62031fd1-034c-4b62-bf54-9a2f06d66278","Type":"ContainerStarted","Data":"c21921b143c83d0b6bdb60dc5aa27fea09728673e04becf8917febd61fbc05a6"} Oct 30 19:15:06.286685 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:06.286630 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" podStartSLOduration=2.286615345 podStartE2EDuration="2.286615345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:06.283747065 +0000 UTC m=+4605.332791981" watchObservedRunningTime="2026-10-30 19:15:06.286615345 +0000 UTC m=+4605.335660259" Oct 30 19:15:07.253324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:07.253256 4862 generic.go:356] "Generic (PLEG): container finished" podID="62031fd1-034c-4b62-bf54-9a2f06d66278" containerID="031e9a9d6788ecf74f0df546c919df87dba1288fe1ee383e341ceeaa1126f84b" exitCode=0 Oct 30 19:15:07.253729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:07.253287 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" event={"ID":"62031fd1-034c-4b62-bf54-9a2f06d66278","Type":"ContainerDied","Data":"031e9a9d6788ecf74f0df546c919df87dba1288fe1ee383e341ceeaa1126f84b"} Oct 30 19:15:08.388723 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.388688 4862 patch_prober.go:29] interesting pod/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:15:08.389178 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.388768 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:15:08.414994 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.414934 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:08.450247 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.450219 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62031fd1-034c-4b62-bf54-9a2f06d66278-kube-api-access-55pt4\" (UniqueName: \"kubernetes.io/projected/62031fd1-034c-4b62-bf54-9a2f06d66278-kube-api-access-55pt4\") pod \"62031fd1-034c-4b62-bf54-9a2f06d66278\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " Oct 30 19:15:08.450381 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.450295 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-xks-ca\") pod \"62031fd1-034c-4b62-bf54-9a2f06d66278\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " Oct 30 19:15:08.450425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.450396 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-openshift-ca\") pod \"62031fd1-034c-4b62-bf54-9a2f06d66278\" (UID: \"62031fd1-034c-4b62-bf54-9a2f06d66278\") " Oct 30 19:15:08.450733 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.450709 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-openshift-ca" pod "62031fd1-034c-4b62-bf54-9a2f06d66278" (UID: "62031fd1-034c-4b62-bf54-9a2f06d66278"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:08.452716 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.452667 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62031fd1-034c-4b62-bf54-9a2f06d66278-kube-api-access-55pt4" pod "62031fd1-034c-4b62-bf54-9a2f06d66278" (UID: "62031fd1-034c-4b62-bf54-9a2f06d66278"). InnerVolumeSpecName "kube-api-access-55pt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:08.454721 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.454695 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-xks-ca" pod "62031fd1-034c-4b62-bf54-9a2f06d66278" (UID: "62031fd1-034c-4b62-bf54-9a2f06d66278"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:08.552315 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.552256 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55pt4\" (UniqueName: \"kubernetes.io/projected/62031fd1-034c-4b62-bf54-9a2f06d66278-kube-api-access-55pt4\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:08.552315 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.552315 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:08.552485 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:08.552326 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62031fd1-034c-4b62-bf54-9a2f06d66278-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.267605 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.267575 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" Oct 30 19:15:09.267789 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.267610 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr" event={"ID":"62031fd1-034c-4b62-bf54-9a2f06d66278","Type":"ContainerDied","Data":"c21921b143c83d0b6bdb60dc5aa27fea09728673e04becf8917febd61fbc05a6"} Oct 30 19:15:09.267789 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.267655 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c21921b143c83d0b6bdb60dc5aa27fea09728673e04becf8917febd61fbc05a6" Oct 30 19:15:09.662408 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.662356 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:15:09.766457 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766430 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-data\") pod \"320d4612-0fc7-411d-8fd7-a1874bf84231\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " Oct 30 19:15:09.766619 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766486 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-token\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-token\") pod \"320d4612-0fc7-411d-8fd7-a1874bf84231\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " Oct 30 19:15:09.766619 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766515 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-podinfo\") pod \"320d4612-0fc7-411d-8fd7-a1874bf84231\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " Oct 30 19:15:09.766619 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766592 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320d4612-0fc7-411d-8fd7-a1874bf84231-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320d4612-0fc7-411d-8fd7-a1874bf84231-istiod-ca-cert\") pod \"320d4612-0fc7-411d-8fd7-a1874bf84231\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " Oct 30 19:15:09.766760 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766624 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-certs\") pod \"320d4612-0fc7-411d-8fd7-a1874bf84231\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " Oct 30 19:15:09.766760 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766728 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-data" pod "320d4612-0fc7-411d-8fd7-a1874bf84231" (UID: "320d4612-0fc7-411d-8fd7-a1874bf84231"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.766867 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766839 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-socket\") pod \"320d4612-0fc7-411d-8fd7-a1874bf84231\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " Oct 30 19:15:09.766938 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766896 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-credential-socket\") pod \"320d4612-0fc7-411d-8fd7-a1874bf84231\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " Oct 30 19:15:09.766938 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766913 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-certs" pod "320d4612-0fc7-411d-8fd7-a1874bf84231" (UID: "320d4612-0fc7-411d-8fd7-a1874bf84231"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.767045 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.766980 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-envoy\") pod \"320d4612-0fc7-411d-8fd7-a1874bf84231\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " Oct 30 19:15:09.767045 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.767031 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-kube-api-access-s2ltj\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-kube-api-access-s2ltj\") pod \"320d4612-0fc7-411d-8fd7-a1874bf84231\" (UID: \"320d4612-0fc7-411d-8fd7-a1874bf84231\") " Oct 30 19:15:09.767149 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.767054 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-socket" pod "320d4612-0fc7-411d-8fd7-a1874bf84231" (UID: "320d4612-0fc7-411d-8fd7-a1874bf84231"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.767149 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.767078 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320d4612-0fc7-411d-8fd7-a1874bf84231-istiod-ca-cert" pod "320d4612-0fc7-411d-8fd7-a1874bf84231" (UID: "320d4612-0fc7-411d-8fd7-a1874bf84231"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:09.767349 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.767197 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-credential-socket" pod "320d4612-0fc7-411d-8fd7-a1874bf84231" (UID: "320d4612-0fc7-411d-8fd7-a1874bf84231"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.767509 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.767493 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.767557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.767512 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320d4612-0fc7-411d-8fd7-a1874bf84231-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.767557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.767521 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.767557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.767530 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.767557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.767538 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.769074 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.769049 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-token" pod "320d4612-0fc7-411d-8fd7-a1874bf84231" (UID: "320d4612-0fc7-411d-8fd7-a1874bf84231"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:09.769445 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.769427 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-kube-api-access-s2ltj" pod "320d4612-0fc7-411d-8fd7-a1874bf84231" (UID: "320d4612-0fc7-411d-8fd7-a1874bf84231"). InnerVolumeSpecName "kube-api-access-s2ltj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:09.769593 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.769574 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-envoy" pod "320d4612-0fc7-411d-8fd7-a1874bf84231" (UID: "320d4612-0fc7-411d-8fd7-a1874bf84231"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:09.769593 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.769579 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-podinfo" pod "320d4612-0fc7-411d-8fd7-a1874bf84231" (UID: "320d4612-0fc7-411d-8fd7-a1874bf84231"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:09.868147 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.868120 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.868147 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.868146 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2ltj\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-kube-api-access-s2ltj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.868332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.868156 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.868332 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.868164 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320d4612-0fc7-411d-8fd7-a1874bf84231-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:09.903007 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.902979 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv"] Oct 30 19:15:09.905700 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.905673 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv"] Oct 30 19:15:09.923635 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:09.923576 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl"] Oct 30 19:15:10.276596 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.276568 4862 generic.go:356] "Generic (PLEG): container finished" podID="8d32fa3e-7c31-403d-bc9d-099927dae6e8" containerID="c4e0f20eb14d79eab4151d1decd58777dc8db503ff3ddcf89223fbbf5c5610e6" exitCode=0 Oct 30 19:15:10.276752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.276639 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" event={"ID":"8d32fa3e-7c31-403d-bc9d-099927dae6e8","Type":"ContainerDied","Data":"c4e0f20eb14d79eab4151d1decd58777dc8db503ff3ddcf89223fbbf5c5610e6"} Oct 30 19:15:10.278442 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.278414 4862 generic.go:356] "Generic (PLEG): container finished" podID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerID="e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd" exitCode=0 Oct 30 19:15:10.278588 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.278485 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" Oct 30 19:15:10.278588 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.278506 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" event={"ID":"320d4612-0fc7-411d-8fd7-a1874bf84231","Type":"ContainerDied","Data":"e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd"} Oct 30 19:15:10.278588 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.278570 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744" event={"ID":"320d4612-0fc7-411d-8fd7-a1874bf84231","Type":"ContainerDied","Data":"9fffcd56a5c2aa663521c61a853865d946e6c6e33fd105e657cc186e1eec7221"} Oct 30 19:15:10.278769 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.278601 4862 scope.go:122] "RemoveContainer" containerID="e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd" Oct 30 19:15:10.280092 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.280066 4862 generic.go:356] "Generic (PLEG): container finished" podID="d3bb768e-e677-44c7-a203-b02125cb3384" containerID="97e686d078905234a44d7d5d989d9e825b2520e102d2513f6f8a5f83e4d82310" exitCode=0 Oct 30 19:15:10.280220 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.280121 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" event={"ID":"d3bb768e-e677-44c7-a203-b02125cb3384","Type":"ContainerDied","Data":"97e686d078905234a44d7d5d989d9e825b2520e102d2513f6f8a5f83e4d82310"} Oct 30 19:15:10.284082 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.284064 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 19:15:10.291507 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.291486 4862 scope.go:122] "RemoveContainer" containerID="e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd" Oct 30 19:15:10.291809 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:10.291782 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd\": container with ID starting with e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd not found: ID does not exist" containerID="e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd" Oct 30 19:15:10.291909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.291816 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd"} err="failed to get container status \"e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd\": rpc error: code = NotFound desc = could not find container \"e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd\": container with ID starting with e63449fdb7e569541855998c2bbd0ebde258ec8194792a1d4180efdae261bbdd not found: ID does not exist" Oct 30 19:15:10.307464 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.307445 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 19:15:10.366425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.366398 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744"] Oct 30 19:15:10.373099 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.373070 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d32fa3e-7c31-403d-bc9d-099927dae6e8-kube-api-access-ch5f7\" (UniqueName: \"kubernetes.io/projected/8d32fa3e-7c31-403d-bc9d-099927dae6e8-kube-api-access-ch5f7\") pod \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\" (UID: \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\") " Oct 30 19:15:10.373233 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.373117 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3bb768e-e677-44c7-a203-b02125cb3384-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bb768e-e677-44c7-a203-b02125cb3384-plugins-config-volume\") pod \"d3bb768e-e677-44c7-a203-b02125cb3384\" (UID: \"d3bb768e-e677-44c7-a203-b02125cb3384\") " Oct 30 19:15:10.373233 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.373157 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d32fa3e-7c31-403d-bc9d-099927dae6e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32fa3e-7c31-403d-bc9d-099927dae6e8-plugins-config-volume\") pod \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\" (UID: \"8d32fa3e-7c31-403d-bc9d-099927dae6e8\") " Oct 30 19:15:10.373481 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.373456 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3bb768e-e677-44c7-a203-b02125cb3384-kube-api-access-l47b8\" (UniqueName: \"kubernetes.io/projected/d3bb768e-e677-44c7-a203-b02125cb3384-kube-api-access-l47b8\") pod \"d3bb768e-e677-44c7-a203-b02125cb3384\" (UID: \"d3bb768e-e677-44c7-a203-b02125cb3384\") " Oct 30 19:15:10.373586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.373534 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3bb768e-e677-44c7-a203-b02125cb3384-plugins-config-volume" pod "d3bb768e-e677-44c7-a203-b02125cb3384" (UID: "d3bb768e-e677-44c7-a203-b02125cb3384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:10.373661 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.373594 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d32fa3e-7c31-403d-bc9d-099927dae6e8-plugins-config-volume" pod "8d32fa3e-7c31-403d-bc9d-099927dae6e8" (UID: "8d32fa3e-7c31-403d-bc9d-099927dae6e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:10.373840 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.373818 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3bb768e-e677-44c7-a203-b02125cb3384-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.373840 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.373841 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d32fa3e-7c31-403d-bc9d-099927dae6e8-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.376002 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.375976 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bb768e-e677-44c7-a203-b02125cb3384-kube-api-access-l47b8" pod "d3bb768e-e677-44c7-a203-b02125cb3384" (UID: "d3bb768e-e677-44c7-a203-b02125cb3384"). InnerVolumeSpecName "kube-api-access-l47b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:10.376112 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.376015 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d32fa3e-7c31-403d-bc9d-099927dae6e8-kube-api-access-ch5f7" pod "8d32fa3e-7c31-403d-bc9d-099927dae6e8" (UID: "8d32fa3e-7c31-403d-bc9d-099927dae6e8"). InnerVolumeSpecName "kube-api-access-ch5f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:10.377307 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.377288 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-09ef1d-openshift-default-65bb585f67-79744"] Oct 30 19:15:10.475318 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.475229 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l47b8\" (UniqueName: \"kubernetes.io/projected/d3bb768e-e677-44c7-a203-b02125cb3384-kube-api-access-l47b8\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.475318 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.475254 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5f7\" (UniqueName: \"kubernetes.io/projected/8d32fa3e-7c31-403d-bc9d-099927dae6e8-kube-api-access-ch5f7\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.748434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.748410 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:15:10.778672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.778641 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-data\") pod \"e7da89a5-2e98-4151-a29a-f350338eca2d\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " Oct 30 19:15:10.778820 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.778743 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-certs\") pod \"e7da89a5-2e98-4151-a29a-f350338eca2d\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " Oct 30 19:15:10.778820 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.778777 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7da89a5-2e98-4151-a29a-f350338eca2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7da89a5-2e98-4151-a29a-f350338eca2d-istiod-ca-cert\") pod \"e7da89a5-2e98-4151-a29a-f350338eca2d\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " Oct 30 19:15:10.778952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.778847 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-kube-api-access-sxtbn\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-kube-api-access-sxtbn\") pod \"e7da89a5-2e98-4151-a29a-f350338eca2d\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " Oct 30 19:15:10.778952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.778893 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-data" pod "e7da89a5-2e98-4151-a29a-f350338eca2d" (UID: "e7da89a5-2e98-4151-a29a-f350338eca2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.778952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.778905 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-podinfo\") pod \"e7da89a5-2e98-4151-a29a-f350338eca2d\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " Oct 30 19:15:10.779107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.778982 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-certs" pod "e7da89a5-2e98-4151-a29a-f350338eca2d" (UID: "e7da89a5-2e98-4151-a29a-f350338eca2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.779107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779002 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-socket\") pod \"e7da89a5-2e98-4151-a29a-f350338eca2d\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " Oct 30 19:15:10.779107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779059 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-token\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-token\") pod \"e7da89a5-2e98-4151-a29a-f350338eca2d\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " Oct 30 19:15:10.779107 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779081 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-credential-socket\") pod \"e7da89a5-2e98-4151-a29a-f350338eca2d\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " Oct 30 19:15:10.779387 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779129 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-envoy\") pod \"e7da89a5-2e98-4151-a29a-f350338eca2d\" (UID: \"e7da89a5-2e98-4151-a29a-f350338eca2d\") " Oct 30 19:15:10.779387 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779288 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-socket" pod "e7da89a5-2e98-4151-a29a-f350338eca2d" (UID: "e7da89a5-2e98-4151-a29a-f350338eca2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.779387 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779311 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7da89a5-2e98-4151-a29a-f350338eca2d-istiod-ca-cert" pod "e7da89a5-2e98-4151-a29a-f350338eca2d" (UID: "e7da89a5-2e98-4151-a29a-f350338eca2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:10.779557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779502 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-credential-socket" pod "e7da89a5-2e98-4151-a29a-f350338eca2d" (UID: "e7da89a5-2e98-4151-a29a-f350338eca2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.779779 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779700 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.779779 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779718 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.779779 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779731 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7da89a5-2e98-4151-a29a-f350338eca2d-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.779779 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779743 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.779779 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.779756 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.781330 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.781299 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-token" pod "e7da89a5-2e98-4151-a29a-f350338eca2d" (UID: "e7da89a5-2e98-4151-a29a-f350338eca2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:10.781330 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.781320 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-podinfo" pod "e7da89a5-2e98-4151-a29a-f350338eca2d" (UID: "e7da89a5-2e98-4151-a29a-f350338eca2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:10.781505 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.781372 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-kube-api-access-sxtbn" pod "e7da89a5-2e98-4151-a29a-f350338eca2d" (UID: "e7da89a5-2e98-4151-a29a-f350338eca2d"). InnerVolumeSpecName "kube-api-access-sxtbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:10.781505 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.781471 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-envoy" pod "e7da89a5-2e98-4151-a29a-f350338eca2d" (UID: "e7da89a5-2e98-4151-a29a-f350338eca2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:10.880818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.880790 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxtbn\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-kube-api-access-sxtbn\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.880818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.880816 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.880818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.880826 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.881057 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.880834 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7da89a5-2e98-4151-a29a-f350338eca2d-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:10.950573 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.950547 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8"] Oct 30 19:15:10.974084 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:10.974057 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx"] Oct 30 19:15:11.070191 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.070157 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" path="/var/lib/kubelet/pods/320d4612-0fc7-411d-8fd7-a1874bf84231/volumes" Oct 30 19:15:11.288487 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.288449 4862 generic.go:356] "Generic (PLEG): container finished" podID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerID="8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6" exitCode=0 Oct 30 19:15:11.288654 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.288514 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" Oct 30 19:15:11.288654 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.288530 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" event={"ID":"e7da89a5-2e98-4151-a29a-f350338eca2d","Type":"ContainerDied","Data":"8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6"} Oct 30 19:15:11.288654 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.288586 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh" event={"ID":"e7da89a5-2e98-4151-a29a-f350338eca2d","Type":"ContainerDied","Data":"e6012321ed93675eedecd86fef3406ac5a139176bd028d6543c5d237c5edfcba"} Oct 30 19:15:11.288654 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.288610 4862 scope.go:122] "RemoveContainer" containerID="8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6" Oct 30 19:15:11.291325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.291308 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" Oct 30 19:15:11.291465 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.291309 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf" event={"ID":"d3bb768e-e677-44c7-a203-b02125cb3384","Type":"ContainerDied","Data":"8efeec6498a59e5c342065e43b9a5b4f8326a96df50e917b787ad3838a47995d"} Oct 30 19:15:11.292828 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.292804 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" event={"ID":"8d32fa3e-7c31-403d-bc9d-099927dae6e8","Type":"ContainerDied","Data":"6df8cee9ade39ae71557994aebb9409a4fc248535ceb4b6870dde8dc9ee4f648"} Oct 30 19:15:11.292993 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.292833 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw" Oct 30 19:15:11.300670 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.300654 4862 scope.go:122] "RemoveContainer" containerID="8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6" Oct 30 19:15:11.300943 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:11.300925 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6\": container with ID starting with 8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6 not found: ID does not exist" containerID="8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6" Oct 30 19:15:11.300992 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.300955 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6"} err="failed to get container status \"8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6\": rpc error: code = NotFound desc = could not find container \"8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6\": container with ID starting with 8f473c6418ed65883796a020f07d91998fae72c7be7ce79a90c8e3be4f7590a6 not found: ID does not exist" Oct 30 19:15:11.300992 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.300973 4862 scope.go:122] "RemoveContainer" containerID="97e686d078905234a44d7d5d989d9e825b2520e102d2513f6f8a5f83e4d82310" Oct 30 19:15:11.309503 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.309483 4862 scope.go:122] "RemoveContainer" containerID="c4e0f20eb14d79eab4151d1decd58777dc8db503ff3ddcf89223fbbf5c5610e6" Oct 30 19:15:11.335589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.335561 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw"] Oct 30 19:15:11.337899 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.337876 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-09ef1d-589d79476c-5snxw"] Oct 30 19:15:11.366351 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.366324 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh"] Oct 30 19:15:11.370435 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.370413 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-c095fa-openshift-default-756895fbf7-jdrgh"] Oct 30 19:15:11.400275 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.400241 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf"] Oct 30 19:15:11.402369 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:11.402351 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-09ef1d-54757db89b-rvhxf"] Oct 30 19:15:13.062532 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:13.062499 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d32fa3e-7c31-403d-bc9d-099927dae6e8" path="/var/lib/kubelet/pods/8d32fa3e-7c31-403d-bc9d-099927dae6e8/volumes" Oct 30 19:15:13.062999 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:13.062871 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3bb768e-e677-44c7-a203-b02125cb3384" path="/var/lib/kubelet/pods/d3bb768e-e677-44c7-a203-b02125cb3384/volumes" Oct 30 19:15:13.063279 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:13.063247 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" path="/var/lib/kubelet/pods/e7da89a5-2e98-4151-a29a-f350338eca2d/volumes" Oct 30 19:15:14.654284 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:14.654226 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" containerID="cri-o://752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65" gracePeriod=45 Oct 30 19:15:14.676434 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:14.676405 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" containerID="cri-o://f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b" gracePeriod=45 Oct 30 19:15:14.730803 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:14.730709 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused" Oct 30 19:15:14.931976 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:14.931953 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 19:15:14.959369 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:14.959345 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 19:15:15.022818 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.022783 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-model-cache\") pod \"d42774cd-c445-4c21-902e-6f832ec412c4\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " Oct 30 19:15:15.023079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.022845 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-home\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-home\") pod \"f0670173-6c92-4b83-913f-946d879df4a1\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " Oct 30 19:15:15.023220 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.022905 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-dshm\") pod \"f0670173-6c92-4b83-913f-946d879df4a1\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " Oct 30 19:15:15.023324 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023253 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-tmp-dir\") pod \"d42774cd-c445-4c21-902e-6f832ec412c4\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " Oct 30 19:15:15.023403 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023062 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-model-cache" pod "d42774cd-c445-4c21-902e-6f832ec412c4" (UID: "d42774cd-c445-4c21-902e-6f832ec412c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.023403 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023378 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-model-cache\") pod \"f0670173-6c92-4b83-913f-946d879df4a1\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " Oct 30 19:15:15.023403 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023107 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-home" pod "f0670173-6c92-4b83-913f-946d879df4a1" (UID: "f0670173-6c92-4b83-913f-946d879df4a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.023586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023410 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-dshm\") pod \"d42774cd-c445-4c21-902e-6f832ec412c4\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " Oct 30 19:15:15.023586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023465 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-tmp-dir" pod "d42774cd-c445-4c21-902e-6f832ec412c4" (UID: "d42774cd-c445-4c21-902e-6f832ec412c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.023586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023473 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-home\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-home\") pod \"d42774cd-c445-4c21-902e-6f832ec412c4\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " Oct 30 19:15:15.023586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023515 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-tmp-dir\") pod \"f0670173-6c92-4b83-913f-946d879df4a1\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " Oct 30 19:15:15.023586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023548 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0670173-6c92-4b83-913f-946d879df4a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0670173-6c92-4b83-913f-946d879df4a1-tls-certs\") pod \"f0670173-6c92-4b83-913f-946d879df4a1\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " Oct 30 19:15:15.023875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023607 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-model-cache" pod "f0670173-6c92-4b83-913f-946d879df4a1" (UID: "f0670173-6c92-4b83-913f-946d879df4a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.023875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023619 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42774cd-c445-4c21-902e-6f832ec412c4-kube-api-access-rrldf\" (UniqueName: \"kubernetes.io/projected/d42774cd-c445-4c21-902e-6f832ec412c4-kube-api-access-rrldf\") pod \"d42774cd-c445-4c21-902e-6f832ec412c4\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " Oct 30 19:15:15.023875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023664 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0670173-6c92-4b83-913f-946d879df4a1-kube-api-access-wdv2s\" (UniqueName: \"kubernetes.io/projected/f0670173-6c92-4b83-913f-946d879df4a1-kube-api-access-wdv2s\") pod \"f0670173-6c92-4b83-913f-946d879df4a1\" (UID: \"f0670173-6c92-4b83-913f-946d879df4a1\") " Oct 30 19:15:15.023875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023720 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d42774cd-c445-4c21-902e-6f832ec412c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/d42774cd-c445-4c21-902e-6f832ec412c4-tls-certs\") pod \"d42774cd-c445-4c21-902e-6f832ec412c4\" (UID: \"d42774cd-c445-4c21-902e-6f832ec412c4\") " Oct 30 19:15:15.023875 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.023800 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-tmp-dir" pod "f0670173-6c92-4b83-913f-946d879df4a1" (UID: "f0670173-6c92-4b83-913f-946d879df4a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.024165 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.024107 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-home" pod "d42774cd-c445-4c21-902e-6f832ec412c4" (UID: "d42774cd-c445-4c21-902e-6f832ec412c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.024235 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.024206 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.024325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.024245 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.024325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.024274 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.024325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.024287 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.024325 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.024301 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.027022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.026994 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-dshm" pod "f0670173-6c92-4b83-913f-946d879df4a1" (UID: "f0670173-6c92-4b83-913f-946d879df4a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.027463 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.027416 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42774cd-c445-4c21-902e-6f832ec412c4-kube-api-access-rrldf" pod "d42774cd-c445-4c21-902e-6f832ec412c4" (UID: "d42774cd-c445-4c21-902e-6f832ec412c4"). InnerVolumeSpecName "kube-api-access-rrldf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:15.027664 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.027640 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0670173-6c92-4b83-913f-946d879df4a1-kube-api-access-wdv2s" pod "f0670173-6c92-4b83-913f-946d879df4a1" (UID: "f0670173-6c92-4b83-913f-946d879df4a1"). InnerVolumeSpecName "kube-api-access-wdv2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:15.028076 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.028050 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0670173-6c92-4b83-913f-946d879df4a1-tls-certs" pod "f0670173-6c92-4b83-913f-946d879df4a1" (UID: "f0670173-6c92-4b83-913f-946d879df4a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:15.028239 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.028218 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-dshm" pod "d42774cd-c445-4c21-902e-6f832ec412c4" (UID: "d42774cd-c445-4c21-902e-6f832ec412c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:15.028775 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.028753 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d42774cd-c445-4c21-902e-6f832ec412c4-tls-certs" pod "d42774cd-c445-4c21-902e-6f832ec412c4" (UID: "d42774cd-c445-4c21-902e-6f832ec412c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:15.125392 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.125360 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0670173-6c92-4b83-913f-946d879df4a1-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.125392 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.125390 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.125595 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.125404 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d42774cd-c445-4c21-902e-6f832ec412c4-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.125595 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.125416 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0670173-6c92-4b83-913f-946d879df4a1-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.125595 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.125431 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrldf\" (UniqueName: \"kubernetes.io/projected/d42774cd-c445-4c21-902e-6f832ec412c4-kube-api-access-rrldf\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.125595 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.125444 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdv2s\" (UniqueName: \"kubernetes.io/projected/f0670173-6c92-4b83-913f-946d879df4a1-kube-api-access-wdv2s\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.125595 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.125458 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d42774cd-c445-4c21-902e-6f832ec412c4-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:15.329554 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.329518 4862 generic.go:356] "Generic (PLEG): container finished" podID="d42774cd-c445-4c21-902e-6f832ec412c4" containerID="752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65" exitCode=0 Oct 30 19:15:15.329752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.329595 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" Oct 30 19:15:15.329752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.329600 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" event={"ID":"d42774cd-c445-4c21-902e-6f832ec412c4","Type":"ContainerDied","Data":"752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65"} Oct 30 19:15:15.329752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.329650 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw" event={"ID":"d42774cd-c445-4c21-902e-6f832ec412c4","Type":"ContainerDied","Data":"df80348762f8b078031d6549487fb9cf77acfdd58351e3dcc21c72e1d4b9ede8"} Oct 30 19:15:15.329752 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.329676 4862 scope.go:122] "RemoveContainer" containerID="752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65" Oct 30 19:15:15.331187 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.331163 4862 generic.go:356] "Generic (PLEG): container finished" podID="f0670173-6c92-4b83-913f-946d879df4a1" containerID="f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b" exitCode=0 Oct 30 19:15:15.331335 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.331200 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" event={"ID":"f0670173-6c92-4b83-913f-946d879df4a1","Type":"ContainerDied","Data":"f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b"} Oct 30 19:15:15.331335 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.331226 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" Oct 30 19:15:15.331335 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.331242 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw" event={"ID":"f0670173-6c92-4b83-913f-946d879df4a1","Type":"ContainerDied","Data":"0aafa6771c89f23bbb595c99724d5fe029ca7ea1b69ecf051cbd6199be19d933"} Oct 30 19:15:15.340571 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.340550 4862 scope.go:122] "RemoveContainer" containerID="752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65" Oct 30 19:15:15.340933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.340911 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz"] Oct 30 19:15:15.341053 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:15.340916 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65\": container with ID starting with 752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65 not found: ID does not exist" containerID="752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65" Oct 30 19:15:15.341053 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341024 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65"} err="failed to get container status \"752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65\": rpc error: code = NotFound desc = could not find container \"752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65\": container with ID starting with 752ccfb0d1267d83c3157a7ef11e2125f9ffd97e12a2e4b132bede4cc2a06a65 not found: ID does not exist" Oct 30 19:15:15.341053 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341041 4862 scope.go:122] "RemoveContainer" containerID="f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b" Oct 30 19:15:15.341539 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341517 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8d32fa3e-7c31-403d-bc9d-099927dae6e8" containerName="payload-pre-processing" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341542 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d32fa3e-7c31-403d-bc9d-099927dae6e8" containerName="payload-pre-processing" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341562 4862 cpu_manager.go:414] "containerMap: removing container" podUID="62031fd1-034c-4b62-bf54-9a2f06d66278" containerName="revoke-keys" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341570 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62031fd1-034c-4b62-bf54-9a2f06d66278" containerName="revoke-keys" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341585 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341592 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341607 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341613 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341623 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341629 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341642 4862 cpu_manager.go:414] "containerMap: removing container" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" Oct 30 19:15:15.341645 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341649 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" Oct 30 19:15:15.342138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341665 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d3bb768e-e677-44c7-a203-b02125cb3384" containerName="payload-processing" Oct 30 19:15:15.342138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341672 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3bb768e-e677-44c7-a203-b02125cb3384" containerName="payload-processing" Oct 30 19:15:15.342138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341762 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" containerName="main" Oct 30 19:15:15.342138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341772 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7da89a5-2e98-4151-a29a-f350338eca2d" containerName="istio-proxy" Oct 30 19:15:15.342138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341778 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="320d4612-0fc7-411d-8fd7-a1874bf84231" containerName="istio-proxy" Oct 30 19:15:15.342138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341784 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3bb768e-e677-44c7-a203-b02125cb3384" containerName="payload-processing" Oct 30 19:15:15.342138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341794 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d32fa3e-7c31-403d-bc9d-099927dae6e8" containerName="payload-pre-processing" Oct 30 19:15:15.342138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341798 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="62031fd1-034c-4b62-bf54-9a2f06d66278" containerName="revoke-keys" Oct 30 19:15:15.342138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.341805 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0670173-6c92-4b83-913f-946d879df4a1" containerName="main" Oct 30 19:15:15.346188 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.346169 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.350619 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.350600 4862 scope.go:122] "RemoveContainer" containerID="f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b" Oct 30 19:15:15.350901 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:15.350880 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b\": container with ID starting with f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b not found: ID does not exist" containerID="f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b" Oct 30 19:15:15.350979 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.350908 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b"} err="failed to get container status \"f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b\": rpc error: code = NotFound desc = could not find container \"f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b\": container with ID starting with f338b7abd20db43b6f74aa9adc42fb4d1c1e818bdc9d7f73c38b8f10cc94542b not found: ID does not exist" Oct 30 19:15:15.360660 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.360639 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:15:15.360771 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.360745 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qccff\"" Oct 30 19:15:15.363797 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.363665 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz"] Oct 30 19:15:15.383408 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.383385 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw"] Oct 30 19:15:15.388362 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.388340 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/distinct-2-w4-fe1374-kserve-df849c688-n6jpw"] Oct 30 19:15:15.430331 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.430305 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46afe4b7-mx7dz\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.430464 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.430438 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46afe4b7-mx7dz\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.430549 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.430503 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xgt\" (UniqueName: \"kubernetes.io/projected/4ad4832f-5acf-4a90-a25d-7991720db89b-kube-api-access-g9xgt\") pod \"maas-api-revoke-sub-46afe4b7-mx7dz\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.452633 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.452608 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw"] Oct 30 19:15:15.457872 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.457847 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fe1374/simulator-w4-fe1374-kserve-784fb8dd45-95qxw"] Oct 30 19:15:15.531166 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.531134 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46afe4b7-mx7dz\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.531329 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.531250 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46afe4b7-mx7dz\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.531405 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.531358 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xgt\" (UniqueName: \"kubernetes.io/projected/4ad4832f-5acf-4a90-a25d-7991720db89b-kube-api-access-g9xgt\") pod \"maas-api-revoke-sub-46afe4b7-mx7dz\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.531816 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.531793 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46afe4b7-mx7dz\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.533858 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.533837 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46afe4b7-mx7dz\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.549965 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.549947 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xgt\" (UniqueName: \"kubernetes.io/projected/4ad4832f-5acf-4a90-a25d-7991720db89b-kube-api-access-g9xgt\") pod \"maas-api-revoke-sub-46afe4b7-mx7dz\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.661151 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.661064 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:15.958382 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:15.958307 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz"] Oct 30 19:15:15.961703 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:15:15.961675 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad4832f_5acf_4a90_a25d_7991720db89b.slice/crio-58f4aed05687b90aaf89377ea28a48a245dd3f8cf3a3508761db6156cfb7c07b WatchSource:0}: Error finding container 58f4aed05687b90aaf89377ea28a48a245dd3f8cf3a3508761db6156cfb7c07b: Status 404 returned error can't find the container with id 58f4aed05687b90aaf89377ea28a48a245dd3f8cf3a3508761db6156cfb7c07b Oct 30 19:15:16.163944 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:16.163912 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz"] Oct 30 19:15:16.341735 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:16.341702 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" event={"ID":"4ad4832f-5acf-4a90-a25d-7991720db89b","Type":"ContainerStarted","Data":"782e9b0a98252e5d7a3825b8cd9b8aec12a2ccf9ed0b0d471a38003cc713c38e"} Oct 30 19:15:16.341932 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:16.341759 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" event={"ID":"4ad4832f-5acf-4a90-a25d-7991720db89b","Type":"ContainerStarted","Data":"58f4aed05687b90aaf89377ea28a48a245dd3f8cf3a3508761db6156cfb7c07b"} Oct 30 19:15:16.376665 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:16.376618 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" podStartSLOduration=1.37660564 podStartE2EDuration="1.37660564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:16.375999634 +0000 UTC m=+4615.425044546" watchObservedRunningTime="2026-10-30 19:15:16.37660564 +0000 UTC m=+4615.425650553" Oct 30 19:15:17.065625 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:17.065588 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42774cd-c445-4c21-902e-6f832ec412c4" path="/var/lib/kubelet/pods/d42774cd-c445-4c21-902e-6f832ec412c4/volumes" Oct 30 19:15:17.066184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:17.066164 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0670173-6c92-4b83-913f-946d879df4a1" path="/var/lib/kubelet/pods/f0670173-6c92-4b83-913f-946d879df4a1/volumes" Oct 30 19:15:17.351225 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:17.351147 4862 generic.go:356] "Generic (PLEG): container finished" podID="4ad4832f-5acf-4a90-a25d-7991720db89b" containerID="782e9b0a98252e5d7a3825b8cd9b8aec12a2ccf9ed0b0d471a38003cc713c38e" exitCode=0 Oct 30 19:15:17.351451 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:17.351217 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" event={"ID":"4ad4832f-5acf-4a90-a25d-7991720db89b","Type":"ContainerDied","Data":"782e9b0a98252e5d7a3825b8cd9b8aec12a2ccf9ed0b0d471a38003cc713c38e"} Oct 30 19:15:18.505821 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.505799 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:18.562073 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.562044 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-openshift-ca\") pod \"4ad4832f-5acf-4a90-a25d-7991720db89b\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " Oct 30 19:15:18.562229 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.562153 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad4832f-5acf-4a90-a25d-7991720db89b-kube-api-access-g9xgt\" (UniqueName: \"kubernetes.io/projected/4ad4832f-5acf-4a90-a25d-7991720db89b-kube-api-access-g9xgt\") pod \"4ad4832f-5acf-4a90-a25d-7991720db89b\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " Oct 30 19:15:18.562318 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.562255 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-xks-ca\") pod \"4ad4832f-5acf-4a90-a25d-7991720db89b\" (UID: \"4ad4832f-5acf-4a90-a25d-7991720db89b\") " Oct 30 19:15:18.562433 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.562405 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-openshift-ca" pod "4ad4832f-5acf-4a90-a25d-7991720db89b" (UID: "4ad4832f-5acf-4a90-a25d-7991720db89b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:18.562731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.562709 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-openshift-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:18.564521 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.564497 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-xks-ca" pod "4ad4832f-5acf-4a90-a25d-7991720db89b" (UID: "4ad4832f-5acf-4a90-a25d-7991720db89b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:18.564623 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.564516 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad4832f-5acf-4a90-a25d-7991720db89b-kube-api-access-g9xgt" pod "4ad4832f-5acf-4a90-a25d-7991720db89b" (UID: "4ad4832f-5acf-4a90-a25d-7991720db89b"). InnerVolumeSpecName "kube-api-access-g9xgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:18.663759 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.663683 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xgt\" (UniqueName: \"kubernetes.io/projected/4ad4832f-5acf-4a90-a25d-7991720db89b-kube-api-access-g9xgt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:18.663759 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:18.663720 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad4832f-5acf-4a90-a25d-7991720db89b-maas-api-xks-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:19.364296 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:19.364196 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" Oct 30 19:15:19.364296 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:19.364220 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46afe4b7-mx7dz" event={"ID":"4ad4832f-5acf-4a90-a25d-7991720db89b","Type":"ContainerDied","Data":"58f4aed05687b90aaf89377ea28a48a245dd3f8cf3a3508761db6156cfb7c07b"} Oct 30 19:15:19.364296 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:19.364280 4862 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58f4aed05687b90aaf89377ea28a48a245dd3f8cf3a3508761db6156cfb7c07b" Oct 30 19:15:19.873371 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:19.873334 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:15:21.164621 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.164582 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" podUID="8eae894d-325c-48b1-bfd3-d016d9d2a741" containerName="payload-processing" containerID="cri-o://ee604b44bbac10caa081bac25f24ecff1b876aa6b59056a9263ce011af3217e8" gracePeriod=25 Oct 30 19:15:21.382236 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.382188 4862 generic.go:356] "Generic (PLEG): container finished" podID="8eae894d-325c-48b1-bfd3-d016d9d2a741" containerID="ee604b44bbac10caa081bac25f24ecff1b876aa6b59056a9263ce011af3217e8" exitCode=0 Oct 30 19:15:21.382408 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.382365 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" event={"ID":"8eae894d-325c-48b1-bfd3-d016d9d2a741","Type":"ContainerDied","Data":"ee604b44bbac10caa081bac25f24ecff1b876aa6b59056a9263ce011af3217e8"} Oct 30 19:15:21.430045 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.429982 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 19:15:21.493687 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.493654 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eae894d-325c-48b1-bfd3-d016d9d2a741-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eae894d-325c-48b1-bfd3-d016d9d2a741-plugins-config-volume\") pod \"8eae894d-325c-48b1-bfd3-d016d9d2a741\" (UID: \"8eae894d-325c-48b1-bfd3-d016d9d2a741\") " Oct 30 19:15:21.493918 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.493710 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eae894d-325c-48b1-bfd3-d016d9d2a741-kube-api-access-92wx6\" (UniqueName: \"kubernetes.io/projected/8eae894d-325c-48b1-bfd3-d016d9d2a741-kube-api-access-92wx6\") pod \"8eae894d-325c-48b1-bfd3-d016d9d2a741\" (UID: \"8eae894d-325c-48b1-bfd3-d016d9d2a741\") " Oct 30 19:15:21.494121 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.494092 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eae894d-325c-48b1-bfd3-d016d9d2a741-plugins-config-volume" pod "8eae894d-325c-48b1-bfd3-d016d9d2a741" (UID: "8eae894d-325c-48b1-bfd3-d016d9d2a741"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:21.496169 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.496138 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eae894d-325c-48b1-bfd3-d016d9d2a741-kube-api-access-92wx6" pod "8eae894d-325c-48b1-bfd3-d016d9d2a741" (UID: "8eae894d-325c-48b1-bfd3-d016d9d2a741"). InnerVolumeSpecName "kube-api-access-92wx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:21.595694 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.595658 4862 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8eae894d-325c-48b1-bfd3-d016d9d2a741-plugins-config-volume\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:21.595694 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:21.595691 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92wx6\" (UniqueName: \"kubernetes.io/projected/8eae894d-325c-48b1-bfd3-d016d9d2a741-kube-api-access-92wx6\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:22.391784 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:22.391747 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" Oct 30 19:15:22.392251 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:22.391780 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz" event={"ID":"8eae894d-325c-48b1-bfd3-d016d9d2a741","Type":"ContainerDied","Data":"91a1885ab9fe194b9f9c8ed6d9adf26408913a7c8d126737de1835fd122eb966"} Oct 30 19:15:22.392251 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:22.391848 4862 scope.go:122] "RemoveContainer" containerID="ee604b44bbac10caa081bac25f24ecff1b876aa6b59056a9263ce011af3217e8" Oct 30 19:15:22.441138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:22.441113 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz"] Oct 30 19:15:22.447307 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:22.447284 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fe1374-65cf8c997d-5j2vz"] Oct 30 19:15:23.060438 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:23.060411 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eae894d-325c-48b1-bfd3-d016d9d2a741" path="/var/lib/kubelet/pods/8eae894d-325c-48b1-bfd3-d016d9d2a741/volumes" Oct 30 19:15:24.181491 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:24.181456 4862 scope.go:122] "RemoveContainer" containerID="a2aeb6f9663f779a3eec7d37370fd4fc3b8f601060ce8b523a8bbb16996e7c3c" Oct 30 19:15:24.190619 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:24.190598 4862 scope.go:122] "RemoveContainer" containerID="26a76ca8a696fa25a13677e649bbc7ccab350a9732deb7e272c00f7f06a849af" Oct 30 19:15:24.199060 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:24.199042 4862 scope.go:122] "RemoveContainer" containerID="6bd005c3a41d0f6cac89fbca033459e910040d56b3740fad49c8e241d6970261" Oct 30 19:15:24.925318 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:24.925248 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" containerID="cri-o://9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d" gracePeriod=45 Oct 30 19:15:24.928714 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:24.928667 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" containerID="cri-o://698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95" gracePeriod=45 Oct 30 19:15:24.943509 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:24.943456 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" containerID="cri-o://dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32" gracePeriod=45 Oct 30 19:15:25.216077 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.216051 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 19:15:25.234731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.234706 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 19:15:25.240815 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.240791 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 19:15:25.331425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331395 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75860e57-2ba4-449e-887c-e8d48a45d9fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/75860e57-2ba4-449e-887c-e8d48a45d9fc-tls-certs\") pod \"75860e57-2ba4-449e-887c-e8d48a45d9fc\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " Oct 30 19:15:25.331425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331437 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tmp-dir\") pod \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " Oct 30 19:15:25.331678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331466 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-dshm\") pod \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " Oct 30 19:15:25.331678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331513 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-home\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-home\") pod \"75860e57-2ba4-449e-887c-e8d48a45d9fc\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " Oct 30 19:15:25.331678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331547 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tls-certs\") pod \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " Oct 30 19:15:25.331678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331572 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tls-certs\") pod \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " Oct 30 19:15:25.331678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331597 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-tmp-dir\") pod \"75860e57-2ba4-449e-887c-e8d48a45d9fc\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " Oct 30 19:15:25.331678 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331653 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75860e57-2ba4-449e-887c-e8d48a45d9fc-kube-api-access-hs6z2\" (UniqueName: \"kubernetes.io/projected/75860e57-2ba4-449e-887c-e8d48a45d9fc-kube-api-access-hs6z2\") pod \"75860e57-2ba4-449e-887c-e8d48a45d9fc\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " Oct 30 19:15:25.331968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331705 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-model-cache\") pod \"75860e57-2ba4-449e-887c-e8d48a45d9fc\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " Oct 30 19:15:25.331968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331718 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tmp-dir" pod "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" (UID: "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.331968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331759 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-home\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-home\") pod \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " Oct 30 19:15:25.331968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331790 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-home" pod "75860e57-2ba4-449e-887c-e8d48a45d9fc" (UID: "75860e57-2ba4-449e-887c-e8d48a45d9fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.331968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331799 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-home\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-home\") pod \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " Oct 30 19:15:25.331968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331892 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-kube-api-access-pvqfb\" (UniqueName: \"kubernetes.io/projected/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-kube-api-access-pvqfb\") pod \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " Oct 30 19:15:25.331968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331920 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-dshm\") pod \"75860e57-2ba4-449e-887c-e8d48a45d9fc\" (UID: \"75860e57-2ba4-449e-887c-e8d48a45d9fc\") " Oct 30 19:15:25.331968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331952 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-kube-api-access-8j2sx\" (UniqueName: \"kubernetes.io/projected/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-kube-api-access-8j2sx\") pod \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " Oct 30 19:15:25.331968 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331943 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-tmp-dir" pod "75860e57-2ba4-449e-887c-e8d48a45d9fc" (UID: "75860e57-2ba4-449e-887c-e8d48a45d9fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.332686 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.331988 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-model-cache\") pod \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " Oct 30 19:15:25.332686 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.332031 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-model-cache\") pod \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " Oct 30 19:15:25.332686 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.332063 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-dshm\") pod \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\" (UID: \"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79\") " Oct 30 19:15:25.332686 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.332139 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tmp-dir\") pod \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\" (UID: \"09fe261a-c563-4a7d-a2c1-6e0c17242fe8\") " Oct 30 19:15:25.332686 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.332607 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-home" pod "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" (UID: "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.332954 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.332778 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.332954 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.332800 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.332954 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.332815 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.332954 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.332828 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.333711 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.333419 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-model-cache" pod "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" (UID: "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.333711 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.333450 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-model-cache" pod "75860e57-2ba4-449e-887c-e8d48a45d9fc" (UID: "75860e57-2ba4-449e-887c-e8d48a45d9fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.333711 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.333463 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tmp-dir" pod "09fe261a-c563-4a7d-a2c1-6e0c17242fe8" (UID: "09fe261a-c563-4a7d-a2c1-6e0c17242fe8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.333711 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.333673 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-home" pod "09fe261a-c563-4a7d-a2c1-6e0c17242fe8" (UID: "09fe261a-c563-4a7d-a2c1-6e0c17242fe8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.333711 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.333688 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-model-cache" pod "09fe261a-c563-4a7d-a2c1-6e0c17242fe8" (UID: "09fe261a-c563-4a7d-a2c1-6e0c17242fe8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.335479 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.335447 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75860e57-2ba4-449e-887c-e8d48a45d9fc-tls-certs" pod "75860e57-2ba4-449e-887c-e8d48a45d9fc" (UID: "75860e57-2ba4-449e-887c-e8d48a45d9fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:25.335583 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.335519 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-dshm" pod "09fe261a-c563-4a7d-a2c1-6e0c17242fe8" (UID: "09fe261a-c563-4a7d-a2c1-6e0c17242fe8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.335891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.335870 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-kube-api-access-pvqfb" pod "09fe261a-c563-4a7d-a2c1-6e0c17242fe8" (UID: "09fe261a-c563-4a7d-a2c1-6e0c17242fe8"). InnerVolumeSpecName "kube-api-access-pvqfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:25.336115 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.336093 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-dshm" pod "75860e57-2ba4-449e-887c-e8d48a45d9fc" (UID: "75860e57-2ba4-449e-887c-e8d48a45d9fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.337219 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.337028 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-kube-api-access-8j2sx" pod "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" (UID: "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79"). InnerVolumeSpecName "kube-api-access-8j2sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:25.337640 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.337621 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-dshm" pod "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" (UID: "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:25.337832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.337802 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tls-certs" pod "09fe261a-c563-4a7d-a2c1-6e0c17242fe8" (UID: "09fe261a-c563-4a7d-a2c1-6e0c17242fe8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:25.337930 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.337829 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75860e57-2ba4-449e-887c-e8d48a45d9fc-kube-api-access-hs6z2" pod "75860e57-2ba4-449e-887c-e8d48a45d9fc" (UID: "75860e57-2ba4-449e-887c-e8d48a45d9fc"). InnerVolumeSpecName "kube-api-access-hs6z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:25.338050 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.338029 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tls-certs" pod "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" (UID: "f92ff043-4e4c-4cb0-b03b-4d13a79a8a79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:25.412729 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.412697 4862 generic.go:356] "Generic (PLEG): container finished" podID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerID="9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d" exitCode=0 Oct 30 19:15:25.412891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.412742 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" Oct 30 19:15:25.412891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.412779 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" event={"ID":"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79","Type":"ContainerDied","Data":"9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d"} Oct 30 19:15:25.412891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.412821 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv" event={"ID":"f92ff043-4e4c-4cb0-b03b-4d13a79a8a79","Type":"ContainerDied","Data":"58044fe1c2a1a65c5fed2fe55679e289af4da8e86bebab27539fb4bef04850cd"} Oct 30 19:15:25.412891 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.412844 4862 scope.go:122] "RemoveContainer" containerID="9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d" Oct 30 19:15:25.414299 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.414237 4862 generic.go:356] "Generic (PLEG): container finished" podID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerID="dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32" exitCode=0 Oct 30 19:15:25.414407 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.414297 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" event={"ID":"09fe261a-c563-4a7d-a2c1-6e0c17242fe8","Type":"ContainerDied","Data":"dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32"} Oct 30 19:15:25.414407 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.414308 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" Oct 30 19:15:25.414407 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.414334 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl" event={"ID":"09fe261a-c563-4a7d-a2c1-6e0c17242fe8","Type":"ContainerDied","Data":"ebf5fb18c1553e0680567575958894a43e190961b0b05f35c02c11bedbd34a27"} Oct 30 19:15:25.416046 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.416021 4862 generic.go:356] "Generic (PLEG): container finished" podID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerID="698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95" exitCode=0 Oct 30 19:15:25.416164 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.416063 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" Oct 30 19:15:25.416164 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.416052 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" event={"ID":"75860e57-2ba4-449e-887c-e8d48a45d9fc","Type":"ContainerDied","Data":"698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95"} Oct 30 19:15:25.416164 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.416100 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv" event={"ID":"75860e57-2ba4-449e-887c-e8d48a45d9fc","Type":"ContainerDied","Data":"63cd1570410283b4d5452f9661e9bece40585f651708c84dab315dba267b68f6"} Oct 30 19:15:25.424621 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.424598 4862 scope.go:122] "RemoveContainer" containerID="9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d" Oct 30 19:15:25.424872 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:25.424854 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d\": container with ID starting with 9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d not found: ID does not exist" containerID="9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d" Oct 30 19:15:25.424940 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.424881 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d"} err="failed to get container status \"9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d\": rpc error: code = NotFound desc = could not find container \"9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d\": container with ID starting with 9f1d8d748ecc535abca5af5004c073918e1664c6520f1df005570eb242d9411d not found: ID does not exist" Oct 30 19:15:25.424940 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.424896 4862 scope.go:122] "RemoveContainer" containerID="dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32" Oct 30 19:15:25.434402 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434378 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434407 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75860e57-2ba4-449e-887c-e8d48a45d9fc-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434421 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434436 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434449 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434462 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs6z2\" (UniqueName: \"kubernetes.io/projected/75860e57-2ba4-449e-887c-e8d48a45d9fc-kube-api-access-hs6z2\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434482 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434476 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434716 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434492 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434716 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434505 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvqfb\" (UniqueName: \"kubernetes.io/projected/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-kube-api-access-pvqfb\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434716 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434519 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75860e57-2ba4-449e-887c-e8d48a45d9fc-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434716 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434534 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j2sx\" (UniqueName: \"kubernetes.io/projected/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-kube-api-access-8j2sx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434716 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434548 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe261a-c563-4a7d-a2c1-6e0c17242fe8-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434716 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434561 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.434716 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434653 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:25.435061 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.434962 4862 scope.go:122] "RemoveContainer" containerID="dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32" Oct 30 19:15:25.435304 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:25.435248 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32\": container with ID starting with dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32 not found: ID does not exist" containerID="dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32" Oct 30 19:15:25.435361 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.435313 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32"} err="failed to get container status \"dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32\": rpc error: code = NotFound desc = could not find container \"dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32\": container with ID starting with dc22a033d5857a4939be908ed603db51f4ea02449a966c4393bb1776bde1ef32 not found: ID does not exist" Oct 30 19:15:25.435361 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.435332 4862 scope.go:122] "RemoveContainer" containerID="698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95" Oct 30 19:15:25.443582 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.443565 4862 scope.go:122] "RemoveContainer" containerID="698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95" Oct 30 19:15:25.443812 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:25.443796 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95\": container with ID starting with 698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95 not found: ID does not exist" containerID="698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95" Oct 30 19:15:25.443854 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.443818 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95"} err="failed to get container status \"698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95\": rpc error: code = NotFound desc = could not find container \"698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95\": container with ID starting with 698dae73b84c2f82eb894de58eb181fb2f6a05a7cd8d8a2de1577708f7815e95 not found: ID does not exist" Oct 30 19:15:25.458843 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.458822 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl"] Oct 30 19:15:25.463527 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.463507 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/simulator-w2-45dc34-kserve-759f94499d-mx4hl"] Oct 30 19:15:25.506078 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.506051 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv"] Oct 30 19:15:25.512033 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.512002 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-2-w2-45dc34-kserve-7d78cd65cc-jtmzv"] Oct 30 19:15:25.541401 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.541379 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv"] Oct 30 19:15:25.545287 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.545257 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-45dc34/distinct-w2-45dc34-kserve-6c88f8597d-5j8zv"] Oct 30 19:15:25.971640 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.971610 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" containerID="cri-o://f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc" gracePeriod=45 Oct 30 19:15:25.994450 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:25.994424 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" containerID="cri-o://70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8" gracePeriod=45 Oct 30 19:15:26.141912 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.141584 4862 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused" Oct 30 19:15:26.250137 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.250109 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 19:15:26.268300 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.268277 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 19:15:26.344504 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344464 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-home\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-home\") pod \"3889cf6b-a606-47e0-a196-879e1b5b30f4\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " Oct 30 19:15:26.344660 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344527 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3889cf6b-a606-47e0-a196-879e1b5b30f4-kube-api-access-dlwzx\" (UniqueName: \"kubernetes.io/projected/3889cf6b-a606-47e0-a196-879e1b5b30f4-kube-api-access-dlwzx\") pod \"3889cf6b-a606-47e0-a196-879e1b5b30f4\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " Oct 30 19:15:26.344660 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344557 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-model-cache\") pod \"4bd52f6f-27c8-415c-aac0-32dad4435204\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " Oct 30 19:15:26.344660 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344593 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-dshm\") pod \"4bd52f6f-27c8-415c-aac0-32dad4435204\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " Oct 30 19:15:26.344660 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344636 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd52f6f-27c8-415c-aac0-32dad4435204-kube-api-access-4hz4h\" (UniqueName: \"kubernetes.io/projected/4bd52f6f-27c8-415c-aac0-32dad4435204-kube-api-access-4hz4h\") pod \"4bd52f6f-27c8-415c-aac0-32dad4435204\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " Oct 30 19:15:26.344919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344676 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3889cf6b-a606-47e0-a196-879e1b5b30f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/3889cf6b-a606-47e0-a196-879e1b5b30f4-tls-certs\") pod \"3889cf6b-a606-47e0-a196-879e1b5b30f4\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " Oct 30 19:15:26.344919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344715 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-model-cache\") pod \"3889cf6b-a606-47e0-a196-879e1b5b30f4\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " Oct 30 19:15:26.344919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344756 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-home" pod "3889cf6b-a606-47e0-a196-879e1b5b30f4" (UID: "3889cf6b-a606-47e0-a196-879e1b5b30f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.344919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344845 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-tmp-dir\") pod \"4bd52f6f-27c8-415c-aac0-32dad4435204\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " Oct 30 19:15:26.344919 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344903 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bd52f6f-27c8-415c-aac0-32dad4435204-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bd52f6f-27c8-415c-aac0-32dad4435204-tls-certs\") pod \"4bd52f6f-27c8-415c-aac0-32dad4435204\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " Oct 30 19:15:26.345209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344949 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-dshm\") pod \"3889cf6b-a606-47e0-a196-879e1b5b30f4\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " Oct 30 19:15:26.345209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.344990 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-home\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-home\") pod \"4bd52f6f-27c8-415c-aac0-32dad4435204\" (UID: \"4bd52f6f-27c8-415c-aac0-32dad4435204\") " Oct 30 19:15:26.345209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345061 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-tmp-dir\") pod \"3889cf6b-a606-47e0-a196-879e1b5b30f4\" (UID: \"3889cf6b-a606-47e0-a196-879e1b5b30f4\") " Oct 30 19:15:26.345209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345131 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-tmp-dir" pod "4bd52f6f-27c8-415c-aac0-32dad4435204" (UID: "4bd52f6f-27c8-415c-aac0-32dad4435204"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.345209 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345200 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-model-cache" pod "4bd52f6f-27c8-415c-aac0-32dad4435204" (UID: "4bd52f6f-27c8-415c-aac0-32dad4435204"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.345500 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345329 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-model-cache" pod "3889cf6b-a606-47e0-a196-879e1b5b30f4" (UID: "3889cf6b-a606-47e0-a196-879e1b5b30f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.345500 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345465 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-tmp-dir" pod "3889cf6b-a606-47e0-a196-879e1b5b30f4" (UID: "3889cf6b-a606-47e0-a196-879e1b5b30f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.345621 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345564 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-home" pod "4bd52f6f-27c8-415c-aac0-32dad4435204" (UID: "4bd52f6f-27c8-415c-aac0-32dad4435204"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.345683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345637 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.345683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345657 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.345683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345669 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.345683 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345682 4862 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-model-cache\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.346074 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.345694 4862 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-tmp-dir\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.348120 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.348097 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd52f6f-27c8-415c-aac0-32dad4435204-tls-certs" pod "4bd52f6f-27c8-415c-aac0-32dad4435204" (UID: "4bd52f6f-27c8-415c-aac0-32dad4435204"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:26.348668 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.348646 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3889cf6b-a606-47e0-a196-879e1b5b30f4-tls-certs" pod "3889cf6b-a606-47e0-a196-879e1b5b30f4" (UID: "3889cf6b-a606-47e0-a196-879e1b5b30f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:26.348668 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.348649 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd52f6f-27c8-415c-aac0-32dad4435204-kube-api-access-4hz4h" pod "4bd52f6f-27c8-415c-aac0-32dad4435204" (UID: "4bd52f6f-27c8-415c-aac0-32dad4435204"). InnerVolumeSpecName "kube-api-access-4hz4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:26.348907 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.348885 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-dshm" pod "3889cf6b-a606-47e0-a196-879e1b5b30f4" (UID: "3889cf6b-a606-47e0-a196-879e1b5b30f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.349004 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.348991 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3889cf6b-a606-47e0-a196-879e1b5b30f4-kube-api-access-dlwzx" pod "3889cf6b-a606-47e0-a196-879e1b5b30f4" (UID: "3889cf6b-a606-47e0-a196-879e1b5b30f4"). InnerVolumeSpecName "kube-api-access-dlwzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:26.350576 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.350555 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-dshm" pod "4bd52f6f-27c8-415c-aac0-32dad4435204" (UID: "4bd52f6f-27c8-415c-aac0-32dad4435204"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:26.423927 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.423898 4862 generic.go:356] "Generic (PLEG): container finished" podID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerID="70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8" exitCode=0 Oct 30 19:15:26.424079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.423945 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" Oct 30 19:15:26.424079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.423971 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" event={"ID":"3889cf6b-a606-47e0-a196-879e1b5b30f4","Type":"ContainerDied","Data":"70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8"} Oct 30 19:15:26.424079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.424005 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx" event={"ID":"3889cf6b-a606-47e0-a196-879e1b5b30f4","Type":"ContainerDied","Data":"c83526fe102a550bde219c9089ac1308ff3d8f29d386752eb6ef348cd323d00c"} Oct 30 19:15:26.424079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.424020 4862 scope.go:122] "RemoveContainer" containerID="70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8" Oct 30 19:15:26.428641 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.428618 4862 generic.go:356] "Generic (PLEG): container finished" podID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerID="f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc" exitCode=0 Oct 30 19:15:26.428740 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.428660 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" event={"ID":"4bd52f6f-27c8-415c-aac0-32dad4435204","Type":"ContainerDied","Data":"f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc"} Oct 30 19:15:26.428740 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.428668 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" Oct 30 19:15:26.428740 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.428684 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8" event={"ID":"4bd52f6f-27c8-415c-aac0-32dad4435204","Type":"ContainerDied","Data":"3d1a8b2ca54351dd8b337e0180a78be6cf01ca5c731cf889be069246ecbf51d8"} Oct 30 19:15:26.436138 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.436119 4862 scope.go:122] "RemoveContainer" containerID="70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8" Oct 30 19:15:26.436432 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:26.436409 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8\": container with ID starting with 70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8 not found: ID does not exist" containerID="70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8" Oct 30 19:15:26.436505 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.436438 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8"} err="failed to get container status \"70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8\": rpc error: code = NotFound desc = could not find container \"70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8\": container with ID starting with 70cc72dfe5d654a7ba29c73cf74bd8aed3ddd73f1c14f6df4885d5d53b1365c8 not found: ID does not exist" Oct 30 19:15:26.436505 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.436454 4862 scope.go:122] "RemoveContainer" containerID="f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc" Oct 30 19:15:26.444855 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.444839 4862 scope.go:122] "RemoveContainer" containerID="f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc" Oct 30 19:15:26.445101 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:26.445076 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc\": container with ID starting with f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc not found: ID does not exist" containerID="f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc" Oct 30 19:15:26.445155 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.445107 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc"} err="failed to get container status \"f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc\": rpc error: code = NotFound desc = could not find container \"f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc\": container with ID starting with f4c55a7da17f7ee79121ac01cd00d4f93ba505e66b7c8b148de83a2f4090fcfc not found: ID does not exist" Oct 30 19:15:26.446589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.446569 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bd52f6f-27c8-415c-aac0-32dad4435204-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.446672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.446592 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3889cf6b-a606-47e0-a196-879e1b5b30f4-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.446672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.446605 4862 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-home\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.446672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.446617 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlwzx\" (UniqueName: \"kubernetes.io/projected/3889cf6b-a606-47e0-a196-879e1b5b30f4-kube-api-access-dlwzx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.446672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.446632 4862 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bd52f6f-27c8-415c-aac0-32dad4435204-dshm\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.446672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.446648 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hz4h\" (UniqueName: \"kubernetes.io/projected/4bd52f6f-27c8-415c-aac0-32dad4435204-kube-api-access-4hz4h\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.446672 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.446660 4862 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3889cf6b-a606-47e0-a196-879e1b5b30f4-tls-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:26.483354 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.483303 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx"] Oct 30 19:15:26.488557 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.488538 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-921450/simulator-w0-921450-kserve-65969d5794-jnqkx"] Oct 30 19:15:26.518895 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.518873 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8"] Oct 30 19:15:26.524312 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:26.524292 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-921450/distinct-w0-921450-kserve-64978d4966-rhpv8"] Oct 30 19:15:27.062311 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:27.062253 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" path="/var/lib/kubelet/pods/09fe261a-c563-4a7d-a2c1-6e0c17242fe8/volumes" Oct 30 19:15:27.062873 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:27.062853 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" path="/var/lib/kubelet/pods/3889cf6b-a606-47e0-a196-879e1b5b30f4/volumes" Oct 30 19:15:27.063240 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:27.063227 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" path="/var/lib/kubelet/pods/4bd52f6f-27c8-415c-aac0-32dad4435204/volumes" Oct 30 19:15:27.064080 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:27.064054 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" path="/var/lib/kubelet/pods/75860e57-2ba4-449e-887c-e8d48a45d9fc/volumes" Oct 30 19:15:27.064508 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:27.064491 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" path="/var/lib/kubelet/pods/f92ff043-4e4c-4cb0-b03b-4d13a79a8a79/volumes" Oct 30 19:15:33.713669 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:33.713590 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt"] Oct 30 19:15:33.714086 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:33.713818 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" podUID="6a1b5627-547f-48aa-b59f-36bcdba7172e" containerName="maas-api" containerID="cri-o://882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed" gracePeriod=30 Oct 30 19:15:33.980475 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:33.980398 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 19:15:34.036029 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.035980 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls\") pod \"6a1b5627-547f-48aa-b59f-36bcdba7172e\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " Oct 30 19:15:34.036233 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.036093 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1b5627-547f-48aa-b59f-36bcdba7172e-kube-api-access-nd6xl\" (UniqueName: \"kubernetes.io/projected/6a1b5627-547f-48aa-b59f-36bcdba7172e-kube-api-access-nd6xl\") pod \"6a1b5627-547f-48aa-b59f-36bcdba7172e\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " Oct 30 19:15:34.036233 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.036154 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1b5627-547f-48aa-b59f-36bcdba7172e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b5627-547f-48aa-b59f-36bcdba7172e-openshift-service-ca\") pod \"6a1b5627-547f-48aa-b59f-36bcdba7172e\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " Oct 30 19:15:34.036395 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.036295 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-metrics-certs\") pod \"6a1b5627-547f-48aa-b59f-36bcdba7172e\" (UID: \"6a1b5627-547f-48aa-b59f-36bcdba7172e\") " Oct 30 19:15:34.036611 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.036584 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1b5627-547f-48aa-b59f-36bcdba7172e-openshift-service-ca" pod "6a1b5627-547f-48aa-b59f-36bcdba7172e" (UID: "6a1b5627-547f-48aa-b59f-36bcdba7172e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:34.036746 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.036727 4862 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b5627-547f-48aa-b59f-36bcdba7172e-openshift-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:34.038299 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.038252 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls" pod "6a1b5627-547f-48aa-b59f-36bcdba7172e" (UID: "6a1b5627-547f-48aa-b59f-36bcdba7172e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:34.038543 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.038522 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1b5627-547f-48aa-b59f-36bcdba7172e-kube-api-access-nd6xl" pod "6a1b5627-547f-48aa-b59f-36bcdba7172e" (UID: "6a1b5627-547f-48aa-b59f-36bcdba7172e"). InnerVolumeSpecName "kube-api-access-nd6xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:34.038637 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.038615 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-metrics-certs" pod "6a1b5627-547f-48aa-b59f-36bcdba7172e" (UID: "6a1b5627-547f-48aa-b59f-36bcdba7172e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:15:34.138157 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.138069 4862 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-maas-api-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:34.138157 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.138104 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd6xl\" (UniqueName: \"kubernetes.io/projected/6a1b5627-547f-48aa-b59f-36bcdba7172e-kube-api-access-nd6xl\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:34.138157 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.138119 4862 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b5627-547f-48aa-b59f-36bcdba7172e-metrics-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:34.436184 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.436149 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr"] Oct 30 19:15:34.442586 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.442559 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-fe1374-2qslr"] Oct 30 19:15:34.489611 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.489583 4862 generic.go:356] "Generic (PLEG): container finished" podID="6a1b5627-547f-48aa-b59f-36bcdba7172e" containerID="882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed" exitCode=0 Oct 30 19:15:34.489731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.489630 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" event={"ID":"6a1b5627-547f-48aa-b59f-36bcdba7172e","Type":"ContainerDied","Data":"882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed"} Oct 30 19:15:34.489731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.489634 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" Oct 30 19:15:34.489731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.489669 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt" event={"ID":"6a1b5627-547f-48aa-b59f-36bcdba7172e","Type":"ContainerDied","Data":"de206d97676d98aee73fd438e7de197c0ee8ee23ba9615c6b301df6368eb7417"} Oct 30 19:15:34.489731 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.489690 4862 scope.go:122] "RemoveContainer" containerID="882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed" Oct 30 19:15:34.499956 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.499940 4862 scope.go:122] "RemoveContainer" containerID="882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed" Oct 30 19:15:34.500213 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:34.500196 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed\": container with ID starting with 882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed not found: ID does not exist" containerID="882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed" Oct 30 19:15:34.500298 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.500221 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed"} err="failed to get container status \"882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed\": rpc error: code = NotFound desc = could not find container \"882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed\": container with ID starting with 882621b40e4fb260b310021c4e34ee3e29381f97177da5d4f03832f26ad891ed not found: ID does not exist" Oct 30 19:15:34.535580 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.535559 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt"] Oct 30 19:15:34.537675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:34.537656 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fe1374-ffb74595b-spqjt"] Oct 30 19:15:35.060952 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:35.060908 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62031fd1-034c-4b62-bf54-9a2f06d66278" path="/var/lib/kubelet/pods/62031fd1-034c-4b62-bf54-9a2f06d66278/volumes" Oct 30 19:15:35.061536 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:35.061514 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1b5627-547f-48aa-b59f-36bcdba7172e" path="/var/lib/kubelet/pods/6a1b5627-547f-48aa-b59f-36bcdba7172e/volumes" Oct 30 19:15:37.945754 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:37.945725 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-rlk29"] Oct 30 19:15:40.008063 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:40.008020 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj"] Oct 30 19:15:40.008508 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:40.008287 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" containerID="cri-o://a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f" gracePeriod=30 Oct 30 19:15:40.081649 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:40.081613 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h"] Oct 30 19:15:40.085225 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:40.085196 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70169aa9-g9k2h"] Oct 30 19:15:41.060715 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:41.060679 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c9d7f7-d869-4565-88d5-44610a2c8b55" path="/var/lib/kubelet/pods/e4c9d7f7-d869-4565-88d5-44610a2c8b55/volumes" Oct 30 19:15:44.009204 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:44.009170 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j"] Oct 30 19:15:44.009808 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:44.009448 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" containerID="cri-o://63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0" gracePeriod=30 Oct 30 19:15:45.003596 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.003558 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5748484d8c-wtztm_8db3b37b-c28b-4b43-8e5d-7e285f8a6819/authorino/0.log" Oct 30 19:15:45.260492 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.260437 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 19:15:45.353005 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.352976 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-certs\") pod \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " Oct 30 19:15:45.353195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353048 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-socket\") pod \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " Oct 30 19:15:45.353195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353074 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istiod-ca-cert\") pod \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " Oct 30 19:15:45.353195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353097 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-token\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-token\") pod \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " Oct 30 19:15:45.353195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353121 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-kube-api-access-mxnmz\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-kube-api-access-mxnmz\") pod \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " Oct 30 19:15:45.353195 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353145 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-data\") pod \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " Oct 30 19:15:45.353531 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353206 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-podinfo\") pod \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " Oct 30 19:15:45.353531 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353289 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-envoy\") pod \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " Oct 30 19:15:45.353531 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353325 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-certs" pod "e6294d21-b8ea-4c03-8d37-9abc5e722b51" (UID: "e6294d21-b8ea-4c03-8d37-9abc5e722b51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:45.353531 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353342 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-credential-socket\") pod \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\" (UID: \"e6294d21-b8ea-4c03-8d37-9abc5e722b51\") " Oct 30 19:15:45.353531 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353355 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-socket" pod "e6294d21-b8ea-4c03-8d37-9abc5e722b51" (UID: "e6294d21-b8ea-4c03-8d37-9abc5e722b51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:45.353778 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353549 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-data" pod "e6294d21-b8ea-4c03-8d37-9abc5e722b51" (UID: "e6294d21-b8ea-4c03-8d37-9abc5e722b51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:45.353778 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353569 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istiod-ca-cert" pod "e6294d21-b8ea-4c03-8d37-9abc5e722b51" (UID: "e6294d21-b8ea-4c03-8d37-9abc5e722b51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:45.353885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353808 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:45.353885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353829 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:45.353885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353842 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:45.353885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.353857 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:45.354283 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.354106 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-credential-socket" pod "e6294d21-b8ea-4c03-8d37-9abc5e722b51" (UID: "e6294d21-b8ea-4c03-8d37-9abc5e722b51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:45.355640 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.355617 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-token" pod "e6294d21-b8ea-4c03-8d37-9abc5e722b51" (UID: "e6294d21-b8ea-4c03-8d37-9abc5e722b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:45.355749 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.355670 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-kube-api-access-mxnmz" pod "e6294d21-b8ea-4c03-8d37-9abc5e722b51" (UID: "e6294d21-b8ea-4c03-8d37-9abc5e722b51"). InnerVolumeSpecName "kube-api-access-mxnmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:45.356093 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.356073 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-envoy" pod "e6294d21-b8ea-4c03-8d37-9abc5e722b51" (UID: "e6294d21-b8ea-4c03-8d37-9abc5e722b51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:45.356157 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.356069 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-podinfo" pod "e6294d21-b8ea-4c03-8d37-9abc5e722b51" (UID: "e6294d21-b8ea-4c03-8d37-9abc5e722b51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:45.455309 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.455256 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:45.455437 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.455310 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxnmz\" (UniqueName: \"kubernetes.io/projected/e6294d21-b8ea-4c03-8d37-9abc5e722b51-kube-api-access-mxnmz\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:45.455437 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.455326 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:45.455437 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.455338 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:45.455437 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.455350 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6294d21-b8ea-4c03-8d37-9abc5e722b51-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:45.553594 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.553567 4862 generic.go:356] "Generic (PLEG): container finished" podID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerID="a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f" exitCode=0 Oct 30 19:15:45.553712 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.553607 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" Oct 30 19:15:45.553712 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.553647 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" event={"ID":"e6294d21-b8ea-4c03-8d37-9abc5e722b51","Type":"ContainerDied","Data":"a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f"} Oct 30 19:15:45.553712 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.553700 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj" event={"ID":"e6294d21-b8ea-4c03-8d37-9abc5e722b51","Type":"ContainerDied","Data":"f21dd93dacaf3b2bddc70f9157719a22d9271f7cedacab70a1fa6640f3a239c0"} Oct 30 19:15:45.553826 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.553720 4862 scope.go:122] "RemoveContainer" containerID="a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f" Oct 30 19:15:45.564481 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.564467 4862 scope.go:122] "RemoveContainer" containerID="a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f" Oct 30 19:15:45.564761 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:45.564727 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f\": container with ID starting with a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f not found: ID does not exist" containerID="a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f" Oct 30 19:15:45.564829 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.564768 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f"} err="failed to get container status \"a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f\": rpc error: code = NotFound desc = could not find container \"a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f\": container with ID starting with a3687f0fcf20a1f455e22e5886e0121f18e830114510f356d114fbdc36864d2f not found: ID does not exist" Oct 30 19:15:45.600420 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.600395 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj"] Oct 30 19:15:45.602161 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:45.602144 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-45dc34-openshift-default-84944d9ff-4jdpj"] Oct 30 19:15:46.693795 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:46.693763 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-rlk29_f3204144-0813-45d0-a202-acb52f0aeaa1/limitador/0.log" Oct 30 19:15:47.061362 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:47.061332 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" path="/var/lib/kubelet/pods/e6294d21-b8ea-4c03-8d37-9abc5e722b51/volumes" Oct 30 19:15:49.263177 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.263153 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 19:15:49.411999 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.411970 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-certs\") pod \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " Oct 30 19:15:49.412202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412047 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istiod-ca-cert\") pod \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " Oct 30 19:15:49.412202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412071 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-credential-socket\") pod \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " Oct 30 19:15:49.412202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412095 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-kube-api-access-nr45m\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-kube-api-access-nr45m\") pod \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " Oct 30 19:15:49.412202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412122 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-socket\") pod \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " Oct 30 19:15:49.412202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412168 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-token\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-token\") pod \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " Oct 30 19:15:49.412540 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412212 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-podinfo\") pod \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " Oct 30 19:15:49.412540 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412252 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-data\") pod \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " Oct 30 19:15:49.412540 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412360 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-certs" pod "ebd3d2ab-2f20-480c-8d11-32012e08b53d" (UID: "ebd3d2ab-2f20-480c-8d11-32012e08b53d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:49.412540 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412365 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-envoy\") pod \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\" (UID: \"ebd3d2ab-2f20-480c-8d11-32012e08b53d\") " Oct 30 19:15:49.412765 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412736 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-credential-socket" pod "ebd3d2ab-2f20-480c-8d11-32012e08b53d" (UID: "ebd3d2ab-2f20-480c-8d11-32012e08b53d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:49.412765 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412750 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-socket" pod "ebd3d2ab-2f20-480c-8d11-32012e08b53d" (UID: "ebd3d2ab-2f20-480c-8d11-32012e08b53d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:49.412870 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412796 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.412944 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.412924 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-data" pod "ebd3d2ab-2f20-480c-8d11-32012e08b53d" (UID: "ebd3d2ab-2f20-480c-8d11-32012e08b53d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:49.413254 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.413233 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istiod-ca-cert" pod "ebd3d2ab-2f20-480c-8d11-32012e08b53d" (UID: "ebd3d2ab-2f20-480c-8d11-32012e08b53d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:49.414924 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.414897 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-token" pod "ebd3d2ab-2f20-480c-8d11-32012e08b53d" (UID: "ebd3d2ab-2f20-480c-8d11-32012e08b53d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:49.415506 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.415441 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-podinfo" pod "ebd3d2ab-2f20-480c-8d11-32012e08b53d" (UID: "ebd3d2ab-2f20-480c-8d11-32012e08b53d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:15:49.415506 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.415467 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-envoy" pod "ebd3d2ab-2f20-480c-8d11-32012e08b53d" (UID: "ebd3d2ab-2f20-480c-8d11-32012e08b53d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:15:49.415784 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.415765 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-kube-api-access-nr45m" pod "ebd3d2ab-2f20-480c-8d11-32012e08b53d" (UID: "ebd3d2ab-2f20-480c-8d11-32012e08b53d"). InnerVolumeSpecName "kube-api-access-nr45m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:49.513850 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.513816 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.513850 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.513843 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.513850 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.513853 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.514085 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.513861 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.514085 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.513870 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr45m\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-kube-api-access-nr45m\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.514085 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.513879 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd3d2ab-2f20-480c-8d11-32012e08b53d-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.514085 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.513887 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.514085 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.513896 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd3d2ab-2f20-480c-8d11-32012e08b53d-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:15:49.579977 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.579930 4862 generic.go:356] "Generic (PLEG): container finished" podID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerID="63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0" exitCode=0 Oct 30 19:15:49.580157 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.580046 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" event={"ID":"ebd3d2ab-2f20-480c-8d11-32012e08b53d","Type":"ContainerDied","Data":"63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0"} Oct 30 19:15:49.580157 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.580058 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" Oct 30 19:15:49.580157 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.580133 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j" event={"ID":"ebd3d2ab-2f20-480c-8d11-32012e08b53d","Type":"ContainerDied","Data":"f68c6f49232c79f1319988d30607f0417e91d3a48ce0550467ef4f147376330e"} Oct 30 19:15:49.580317 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.580161 4862 scope.go:122] "RemoveContainer" containerID="63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0" Oct 30 19:15:49.597810 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.597787 4862 scope.go:122] "RemoveContainer" containerID="63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0" Oct 30 19:15:49.598201 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:49.598169 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0\": container with ID starting with 63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0 not found: ID does not exist" containerID="63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0" Oct 30 19:15:49.598333 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.598210 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0"} err="failed to get container status \"63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0\": rpc error: code = NotFound desc = could not find container \"63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0\": container with ID starting with 63ebb9b39c17c2b3fa1cd26c45a8b75dfe202ebffd8f6c78a472219387a5bae0 not found: ID does not exist" Oct 30 19:15:49.632425 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.632394 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j"] Oct 30 19:15:49.636111 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:49.636077 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-921450-openshift-default-7b67445fff-vq47j"] Oct 30 19:15:49.869391 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:15:49.869352 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:15:51.060863 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:15:51.060831 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" path="/var/lib/kubelet/pods/ebd3d2ab-2f20-480c-8d11-32012e08b53d/volumes" Oct 30 19:16:00.089897 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:00.089867 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/1.log" Oct 30 19:16:01.376388 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:01.376356 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft"] Oct 30 19:16:01.376793 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:01.376553 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" containerID="cri-o://50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329" gracePeriod=30 Oct 30 19:16:01.387329 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:01.387301 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4"] Oct 30 19:16:01.387506 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:01.387487 4862 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" containerID="cri-o://173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367" gracePeriod=30 Oct 30 19:16:06.637432 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.637405 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:16:06.661006 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.660947 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-data\") pod \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " Oct 30 19:16:06.661006 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.660992 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-credential-socket\") pod \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " Oct 30 19:16:06.661202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661048 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-certs\") pod \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " Oct 30 19:16:06.661202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661098 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-socket\") pod \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " Oct 30 19:16:06.661202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661133 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-kube-api-access-gdktf\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-kube-api-access-gdktf\") pod \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " Oct 30 19:16:06.661202 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661183 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-data" pod "d170b4b8-91b7-4857-bcbf-0f28ccb608bc" (UID: "d170b4b8-91b7-4857-bcbf-0f28ccb608bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.661480 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661212 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-token\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-token\") pod \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " Oct 30 19:16:06.661480 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661294 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-podinfo\") pod \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " Oct 30 19:16:06.661480 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661316 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-certs" pod "d170b4b8-91b7-4857-bcbf-0f28ccb608bc" (UID: "d170b4b8-91b7-4857-bcbf-0f28ccb608bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.661480 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661356 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-envoy\") pod \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " Oct 30 19:16:06.661480 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661402 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istiod-ca-cert\") pod \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\" (UID: \"d170b4b8-91b7-4857-bcbf-0f28ccb608bc\") " Oct 30 19:16:06.661480 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661470 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:16:06.661790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661471 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-credential-socket" pod "d170b4b8-91b7-4857-bcbf-0f28ccb608bc" (UID: "d170b4b8-91b7-4857-bcbf-0f28ccb608bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.661790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661483 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-socket" pod "d170b4b8-91b7-4857-bcbf-0f28ccb608bc" (UID: "d170b4b8-91b7-4857-bcbf-0f28ccb608bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.661790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661714 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.661790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661732 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.661790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661746 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.661790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661760 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.662099 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.661849 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istiod-ca-cert" pod "d170b4b8-91b7-4857-bcbf-0f28ccb608bc" (UID: "d170b4b8-91b7-4857-bcbf-0f28ccb608bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:06.663595 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.663576 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-envoy" pod "d170b4b8-91b7-4857-bcbf-0f28ccb608bc" (UID: "d170b4b8-91b7-4857-bcbf-0f28ccb608bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.663684 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.663670 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-kube-api-access-gdktf" pod "d170b4b8-91b7-4857-bcbf-0f28ccb608bc" (UID: "d170b4b8-91b7-4857-bcbf-0f28ccb608bc"). InnerVolumeSpecName "kube-api-access-gdktf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:06.664045 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.664018 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-token" pod "d170b4b8-91b7-4857-bcbf-0f28ccb608bc" (UID: "d170b4b8-91b7-4857-bcbf-0f28ccb608bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:06.664045 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.664034 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-podinfo" pod "d170b4b8-91b7-4857-bcbf-0f28ccb608bc" (UID: "d170b4b8-91b7-4857-bcbf-0f28ccb608bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:16:06.679695 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.679666 4862 generic.go:356] "Generic (PLEG): container finished" podID="7329398d-3b90-4434-ad23-3429cb00ec52" containerID="173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367" exitCode=0 Oct 30 19:16:06.679794 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.679702 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" Oct 30 19:16:06.679794 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.679699 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" event={"ID":"7329398d-3b90-4434-ad23-3429cb00ec52","Type":"ContainerDied","Data":"173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367"} Oct 30 19:16:06.679916 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.679822 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4" event={"ID":"7329398d-3b90-4434-ad23-3429cb00ec52","Type":"ContainerDied","Data":"64cb1f2559dadd9915c2398f2f04f0f9995f262d323a5438bb66dc475c5b55ac"} Oct 30 19:16:06.679916 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.679858 4862 scope.go:122] "RemoveContainer" containerID="173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367" Oct 30 19:16:06.681183 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.681165 4862 generic.go:356] "Generic (PLEG): container finished" podID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerID="50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329" exitCode=0 Oct 30 19:16:06.681303 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.681201 4862 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" Oct 30 19:16:06.681303 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.681242 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" event={"ID":"d170b4b8-91b7-4857-bcbf-0f28ccb608bc","Type":"ContainerDied","Data":"50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329"} Oct 30 19:16:06.681422 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.681313 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft" event={"ID":"d170b4b8-91b7-4857-bcbf-0f28ccb608bc","Type":"ContainerDied","Data":"6e8befd1c6b344b64a5e81503136dd9c8839c010afb599e77d9bc9453c5d1368"} Oct 30 19:16:06.691286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.691240 4862 scope.go:122] "RemoveContainer" containerID="173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367" Oct 30 19:16:06.691647 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:16:06.691600 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367\": container with ID starting with 173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367 not found: ID does not exist" containerID="173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367" Oct 30 19:16:06.691733 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.691652 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367"} err="failed to get container status \"173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367\": rpc error: code = NotFound desc = could not find container \"173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367\": container with ID starting with 173da9a0bf2f554e11dd3cb7e2eb1e23be3e65faa1bdb4baf5b4280867eb3367 not found: ID does not exist" Oct 30 19:16:06.691733 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.691671 4862 scope.go:122] "RemoveContainer" containerID="50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329" Oct 30 19:16:06.701992 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.701969 4862 scope.go:122] "RemoveContainer" containerID="50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329" Oct 30 19:16:06.702293 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:16:06.702251 4862 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329\": container with ID starting with 50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329 not found: ID does not exist" containerID="50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329" Oct 30 19:16:06.702380 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.702304 4862 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329"} err="failed to get container status \"50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329\": rpc error: code = NotFound desc = could not find container \"50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329\": container with ID starting with 50585f2559f4e0ebe5fddbc132cb44f208fc4c8465fec0fb9ec202e5b2f75329 not found: ID does not exist" Oct 30 19:16:06.762358 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762336 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-envoy\") pod \"7329398d-3b90-4434-ad23-3429cb00ec52\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " Oct 30 19:16:06.762511 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762374 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-istio-token\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-istio-token\") pod \"7329398d-3b90-4434-ad23-3429cb00ec52\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " Oct 30 19:16:06.762511 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762404 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7329398d-3b90-4434-ad23-3429cb00ec52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7329398d-3b90-4434-ad23-3429cb00ec52-istiod-ca-cert\") pod \"7329398d-3b90-4434-ad23-3429cb00ec52\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " Oct 30 19:16:06.762511 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762445 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-kube-api-access-fkfcj\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-kube-api-access-fkfcj\") pod \"7329398d-3b90-4434-ad23-3429cb00ec52\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " Oct 30 19:16:06.762511 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762492 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-certs\") pod \"7329398d-3b90-4434-ad23-3429cb00ec52\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " Oct 30 19:16:06.762781 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762558 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7329398d-3b90-4434-ad23-3429cb00ec52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7329398d-3b90-4434-ad23-3429cb00ec52-istio-podinfo\") pod \"7329398d-3b90-4434-ad23-3429cb00ec52\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " Oct 30 19:16:06.762781 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762610 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-credential-socket\") pod \"7329398d-3b90-4434-ad23-3429cb00ec52\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " Oct 30 19:16:06.762781 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762660 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-data\") pod \"7329398d-3b90-4434-ad23-3429cb00ec52\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " Oct 30 19:16:06.762781 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762691 4862 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-socket\") pod \"7329398d-3b90-4434-ad23-3429cb00ec52\" (UID: \"7329398d-3b90-4434-ad23-3429cb00ec52\") " Oct 30 19:16:06.763022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762795 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-certs" pod "7329398d-3b90-4434-ad23-3429cb00ec52" (UID: "7329398d-3b90-4434-ad23-3429cb00ec52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.763022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762876 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7329398d-3b90-4434-ad23-3429cb00ec52-istiod-ca-cert" pod "7329398d-3b90-4434-ad23-3429cb00ec52" (UID: "7329398d-3b90-4434-ad23-3429cb00ec52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:16:06.763022 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.762997 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-socket" pod "7329398d-3b90-4434-ad23-3429cb00ec52" (UID: "7329398d-3b90-4434-ad23-3429cb00ec52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.763675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.763043 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdktf\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-kube-api-access-gdktf\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.763675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.763063 4862 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.763675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.763078 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.763675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.763092 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.763675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.763105 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.763675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.763106 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-credential-socket" pod "7329398d-3b90-4434-ad23-3429cb00ec52" (UID: "7329398d-3b90-4434-ad23-3429cb00ec52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.763675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.763118 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d170b4b8-91b7-4857-bcbf-0f28ccb608bc-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.763675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.763132 4862 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7329398d-3b90-4434-ad23-3429cb00ec52-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.763675 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.763381 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-data" pod "7329398d-3b90-4434-ad23-3429cb00ec52" (UID: "7329398d-3b90-4434-ad23-3429cb00ec52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.764701 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.764680 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-envoy" pod "7329398d-3b90-4434-ad23-3429cb00ec52" (UID: "7329398d-3b90-4434-ad23-3429cb00ec52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:16:06.764791 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.764745 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-kube-api-access-fkfcj" pod "7329398d-3b90-4434-ad23-3429cb00ec52" (UID: "7329398d-3b90-4434-ad23-3429cb00ec52"). InnerVolumeSpecName "kube-api-access-fkfcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:06.764983 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.764968 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-istio-token" pod "7329398d-3b90-4434-ad23-3429cb00ec52" (UID: "7329398d-3b90-4434-ad23-3429cb00ec52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:16:06.765125 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.765108 4862 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7329398d-3b90-4434-ad23-3429cb00ec52-istio-podinfo" pod "7329398d-3b90-4434-ad23-3429cb00ec52" (UID: "7329398d-3b90-4434-ad23-3429cb00ec52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:16:06.769052 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.769032 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft"] Oct 30 19:16:06.772368 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.772350 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-dxjft"] Oct 30 19:16:06.863570 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.863545 4862 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7329398d-3b90-4434-ad23-3429cb00ec52-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.863570 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.863568 4862 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.863738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.863578 4862 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.863738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.863587 4862 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.863738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.863595 4862 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7329398d-3b90-4434-ad23-3429cb00ec52-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.863738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.863603 4862 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:06.863738 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:06.863611 4862 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkfcj\" (UniqueName: \"kubernetes.io/projected/7329398d-3b90-4434-ad23-3429cb00ec52-kube-api-access-fkfcj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\"" Oct 30 19:16:07.021499 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:07.021436 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4"] Oct 30 19:16:07.031847 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:07.031824 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76597d6855-7lhc4"] Oct 30 19:16:07.060370 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:07.060345 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" path="/var/lib/kubelet/pods/7329398d-3b90-4434-ad23-3429cb00ec52/volumes" Oct 30 19:16:07.060778 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:07.060765 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" path="/var/lib/kubelet/pods/d170b4b8-91b7-4857-bcbf-0f28ccb608bc/volumes" Oct 30 19:16:10.951275 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:10.951222 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9jr6v_e7d0e911-850b-46bc-aabe-237660e9dbaa/manager/0.log" Oct 30 19:16:19.869785 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:16:19.869750 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:16:21.217088 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:21.217037 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7rt9f_7afa1539-e989-4bdd-bdda-eccf8ae776c3/proxy/0.log" Oct 30 19:16:21.961485 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:21.961455 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/0.log" Oct 30 19:16:22.842655 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:22.842626 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5748484d8c-wtztm_8db3b37b-c28b-4b43-8e5d-7e285f8a6819/authorino/0.log" Oct 30 19:16:24.267008 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:24.266977 4862 scope.go:122] "RemoveContainer" containerID="63e9274bc986f5f12a92edcfb1cd946ab29ac0e40388668ccd4bba12de6c8274" Oct 30 19:16:24.275860 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:24.275842 4862 scope.go:122] "RemoveContainer" containerID="bd83d93ff6f4dec0b917389dc1a55521b35d05f9f944b4e0c6e002668cfc3b41" Oct 30 19:16:24.283759 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:24.283741 4862 scope.go:122] "RemoveContainer" containerID="6e81277e82de9c095b38973252f37566bd508e23ee312342dc9add3bdf65bd32" Oct 30 19:16:44.381690 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:44.381655 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/1.log" Oct 30 19:16:45.473197 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:45.473168 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9jr6v_e7d0e911-850b-46bc-aabe-237660e9dbaa/manager/0.log" Oct 30 19:16:45.711773 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:45.711748 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7rt9f_7afa1539-e989-4bdd-bdda-eccf8ae776c3/proxy/0.log" Oct 30 19:16:46.355762 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:46.355734 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/0.log" Oct 30 19:16:47.210684 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:47.210658 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b4d8998bc-4lr55_7254fe79-25df-49c3-843c-51f10d54f5f4/maas-api/0.log" Oct 30 19:16:47.447902 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:47.447878 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-009f69e1-pt2vc_49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c/revoke-keys/0.log" Oct 30 19:16:47.649566 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:47.649542 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0a615556-ccrzt_bbb9bd28-af5e-40a7-985f-509d0bded58c/revoke-keys/0.log" Oct 30 19:16:47.831160 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:47.831132 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-46afe4b7-mx7dz_4ad4832f-5acf-4a90-a25d-7991720db89b/revoke-keys/0.log" Oct 30 19:16:48.035320 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:48.035246 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5f7418fe-5mk5m_62015e36-11c0-4b1e-ba85-d76c99717575/revoke-keys/0.log" Oct 30 19:16:48.804272 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:48.804233 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8b0ee064-scspg_0eaffcd7-2ea0-4a5c-ad87-90a65c976251/revoke-keys/0.log" Oct 30 19:16:48.987975 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:48.987949 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-af7176ea-rgcwt_ae077ad8-9e07-44f1-86a5-b0c97096e287/revoke-keys/0.log" Oct 30 19:16:49.186129 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:49.186058 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b1120e8d-d2bf5_d65b1b3b-b94c-457c-ab38-712fa4ad7d13/revoke-keys/0.log" Oct 30 19:16:49.583000 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:49.582978 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d9f7c159-6q4ff_bf1700b0-10fd-4357-96b4-f665910f6ebb/revoke-keys/0.log" Oct 30 19:16:49.868761 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:16:49.868682 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:16:51.670174 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:51.670144 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5748484d8c-wtztm_8db3b37b-c28b-4b43-8e5d-7e285f8a6819/authorino/0.log" Oct 30 19:16:53.257674 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:53.257638 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-rlk29_f3204144-0813-45d0-a202-acb52f0aeaa1/limitador/0.log" Oct 30 19:16:55.599098 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:55.599065 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc6d98d7-wrwtb_c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e/kube-auth-proxy/0.log" Oct 30 19:16:55.788141 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:55.788112 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-46d6l_6318be39-5955-4db1-99f2-318432ad0e0d/istio-proxy/0.log" Oct 30 19:16:56.026618 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:56.026543 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg"] Oct 30 19:16:56.036496 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:56.036474 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b0ee064-scspg"] Oct 30 19:16:56.956499 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:56.956460 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ff9667845-fjrpd_5b5e199e-2896-450f-9e73-838fb7e12e48/router/0.log" Oct 30 19:16:57.060329 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:57.060301 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eaffcd7-2ea0-4a5c-ad87-90a65c976251" path="/var/lib/kubelet/pods/0eaffcd7-2ea0-4a5c-ad87-90a65c976251/volumes" Oct 30 19:16:59.075721 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:16:59.075682 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-7dkfn_da01fa56-122a-490d-999b-e2c20b957e4e/main/0.log" Oct 30 19:17:06.030384 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:06.030306 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt"] Oct 30 19:17:06.040299 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:06.040275 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af7176ea-rgcwt"] Oct 30 19:17:07.061499 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:07.061465 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae077ad8-9e07-44f1-86a5-b0c97096e287" path="/var/lib/kubelet/pods/ae077ad8-9e07-44f1-86a5-b0c97096e287/volumes" Oct 30 19:17:18.049765 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:18.049730 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc"] Oct 30 19:17:18.061256 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:18.061221 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-009f69e1-pt2vc"] Oct 30 19:17:19.059959 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.059924 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c" path="/var/lib/kubelet/pods/49a4ecb7-ec5c-4e2e-84a1-ffd158ad6c1c/volumes" Oct 30 19:17:19.261528 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261497 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p4hc2/must-gather-7qp95"] Oct 30 19:17:19.261847 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261833 4862 cpu_manager.go:414] "containerMap: removing container" podUID="8eae894d-325c-48b1-bfd3-d016d9d2a741" containerName="payload-processing" Oct 30 19:17:19.261893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261848 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eae894d-325c-48b1-bfd3-d016d9d2a741" containerName="payload-processing" Oct 30 19:17:19.261893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261861 4862 cpu_manager.go:414] "containerMap: removing container" podUID="4ad4832f-5acf-4a90-a25d-7991720db89b" containerName="revoke-keys" Oct 30 19:17:19.261893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261866 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad4832f-5acf-4a90-a25d-7991720db89b" containerName="revoke-keys" Oct 30 19:17:19.261893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261873 4862 cpu_manager.go:414] "containerMap: removing container" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" Oct 30 19:17:19.261893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261878 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" Oct 30 19:17:19.261893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261887 4862 cpu_manager.go:414] "containerMap: removing container" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" Oct 30 19:17:19.261893 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261894 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261902 4862 cpu_manager.go:414] "containerMap: removing container" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261906 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261916 4862 cpu_manager.go:414] "containerMap: removing container" podUID="6a1b5627-547f-48aa-b59f-36bcdba7172e" containerName="maas-api" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261921 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1b5627-547f-48aa-b59f-36bcdba7172e" containerName="maas-api" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261927 4862 cpu_manager.go:414] "containerMap: removing container" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261932 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261940 4862 cpu_manager.go:414] "containerMap: removing container" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261944 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261955 4862 cpu_manager.go:414] "containerMap: removing container" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261959 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261965 4862 cpu_manager.go:414] "containerMap: removing container" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261969 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261978 4862 cpu_manager.go:414] "containerMap: removing container" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261982 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261995 4862 cpu_manager.go:414] "containerMap: removing container" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.261999 4862 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262056 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1b5627-547f-48aa-b59f-36bcdba7172e" containerName="maas-api" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262064 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="f92ff043-4e4c-4cb0-b03b-4d13a79a8a79" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262069 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="3889cf6b-a606-47e0-a196-879e1b5b30f4" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262077 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6294d21-b8ea-4c03-8d37-9abc5e722b51" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262083 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="75860e57-2ba4-449e-887c-e8d48a45d9fc" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262088 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="7329398d-3b90-4434-ad23-3429cb00ec52" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262095 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd3d2ab-2f20-480c-8d11-32012e08b53d" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262102 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eae894d-325c-48b1-bfd3-d016d9d2a741" containerName="payload-processing" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262107 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad4832f-5acf-4a90-a25d-7991720db89b" containerName="revoke-keys" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262112 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="d170b4b8-91b7-4857-bcbf-0f28ccb608bc" containerName="istio-proxy" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262117 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd52f6f-27c8-415c-aac0-32dad4435204" containerName="main" Oct 30 19:17:19.262118 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.262121 4862 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fe261a-c563-4a7d-a2c1-6e0c17242fe8" containerName="main" Oct 30 19:17:19.265688 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.265670 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p4hc2/must-gather-7qp95" Oct 30 19:17:19.297035 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.297005 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p4hc2\"/\"openshift-service-ca.crt\"" Oct 30 19:17:19.297182 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.297168 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p4hc2\"/\"kube-root-ca.crt\"" Oct 30 19:17:19.298536 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.298519 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p4hc2\"/\"default-dockercfg-8p84j\"" Oct 30 19:17:19.315471 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.315421 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p4hc2/must-gather-7qp95"] Oct 30 19:17:19.338876 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.338852 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/83cd0df9-68c5-4c3f-abbc-14aebea8e91c-must-gather-output\") pod \"must-gather-7qp95\" (UID: \"83cd0df9-68c5-4c3f-abbc-14aebea8e91c\") " pod="openshift-must-gather-p4hc2/must-gather-7qp95" Oct 30 19:17:19.338992 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.338911 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8j4p\" (UniqueName: \"kubernetes.io/projected/83cd0df9-68c5-4c3f-abbc-14aebea8e91c-kube-api-access-d8j4p\") pod \"must-gather-7qp95\" (UID: \"83cd0df9-68c5-4c3f-abbc-14aebea8e91c\") " pod="openshift-must-gather-p4hc2/must-gather-7qp95" Oct 30 19:17:19.417253 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.417219 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wj67d/must-gather-fg7h2"] Oct 30 19:17:19.420845 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.420825 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj67d/must-gather-fg7h2" Oct 30 19:17:19.439416 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.439391 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8d0b230-a84e-4814-86bb-89794ed809b5-must-gather-output\") pod \"must-gather-fg7h2\" (UID: \"d8d0b230-a84e-4814-86bb-89794ed809b5\") " pod="openshift-must-gather-wj67d/must-gather-fg7h2" Oct 30 19:17:19.439543 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.439421 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7nh\" (UniqueName: \"kubernetes.io/projected/d8d0b230-a84e-4814-86bb-89794ed809b5-kube-api-access-7l7nh\") pod \"must-gather-fg7h2\" (UID: \"d8d0b230-a84e-4814-86bb-89794ed809b5\") " pod="openshift-must-gather-wj67d/must-gather-fg7h2" Oct 30 19:17:19.439543 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.439451 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/83cd0df9-68c5-4c3f-abbc-14aebea8e91c-must-gather-output\") pod \"must-gather-7qp95\" (UID: \"83cd0df9-68c5-4c3f-abbc-14aebea8e91c\") " pod="openshift-must-gather-p4hc2/must-gather-7qp95" Oct 30 19:17:19.439695 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.439607 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8j4p\" (UniqueName: \"kubernetes.io/projected/83cd0df9-68c5-4c3f-abbc-14aebea8e91c-kube-api-access-d8j4p\") pod \"must-gather-7qp95\" (UID: \"83cd0df9-68c5-4c3f-abbc-14aebea8e91c\") " pod="openshift-must-gather-p4hc2/must-gather-7qp95" Oct 30 19:17:19.439800 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.439776 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/83cd0df9-68c5-4c3f-abbc-14aebea8e91c-must-gather-output\") pod \"must-gather-7qp95\" (UID: \"83cd0df9-68c5-4c3f-abbc-14aebea8e91c\") " pod="openshift-must-gather-p4hc2/must-gather-7qp95" Oct 30 19:17:19.448091 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.448062 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj67d/must-gather-fg7h2"] Oct 30 19:17:19.456500 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.456482 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj67d\"/\"openshift-service-ca.crt\"" Oct 30 19:17:19.456610 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.456520 4862 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj67d\"/\"kube-root-ca.crt\"" Oct 30 19:17:19.462933 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.462913 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8j4p\" (UniqueName: \"kubernetes.io/projected/83cd0df9-68c5-4c3f-abbc-14aebea8e91c-kube-api-access-d8j4p\") pod \"must-gather-7qp95\" (UID: \"83cd0df9-68c5-4c3f-abbc-14aebea8e91c\") " pod="openshift-must-gather-p4hc2/must-gather-7qp95" Oct 30 19:17:19.537470 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.537445 4862 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wj67d\"/\"default-dockercfg-g4gj2\"" Oct 30 19:17:19.540883 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.540863 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8d0b230-a84e-4814-86bb-89794ed809b5-must-gather-output\") pod \"must-gather-fg7h2\" (UID: \"d8d0b230-a84e-4814-86bb-89794ed809b5\") " pod="openshift-must-gather-wj67d/must-gather-fg7h2" Oct 30 19:17:19.540984 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.540907 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7nh\" (UniqueName: \"kubernetes.io/projected/d8d0b230-a84e-4814-86bb-89794ed809b5-kube-api-access-7l7nh\") pod \"must-gather-fg7h2\" (UID: \"d8d0b230-a84e-4814-86bb-89794ed809b5\") " pod="openshift-must-gather-wj67d/must-gather-fg7h2" Oct 30 19:17:19.541204 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.541188 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8d0b230-a84e-4814-86bb-89794ed809b5-must-gather-output\") pod \"must-gather-fg7h2\" (UID: \"d8d0b230-a84e-4814-86bb-89794ed809b5\") " pod="openshift-must-gather-wj67d/must-gather-fg7h2" Oct 30 19:17:19.562909 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.562886 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7nh\" (UniqueName: \"kubernetes.io/projected/d8d0b230-a84e-4814-86bb-89794ed809b5-kube-api-access-7l7nh\") pod \"must-gather-fg7h2\" (UID: \"d8d0b230-a84e-4814-86bb-89794ed809b5\") " pod="openshift-must-gather-wj67d/must-gather-fg7h2" Oct 30 19:17:19.579790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.579741 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p4hc2/must-gather-7qp95" Oct 30 19:17:19.734280 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.734228 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj67d/must-gather-fg7h2" Oct 30 19:17:19.869612 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:17:19.869537 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:17:19.901228 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.901191 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p4hc2/must-gather-7qp95"] Oct 30 19:17:19.905209 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:17:19.905184 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83cd0df9_68c5_4c3f_abbc_14aebea8e91c.slice/crio-4fb02a212d15e9895caccfafe92baed160e0bcad0cf4da771c61a75007d43f18 WatchSource:0}: Error finding container 4fb02a212d15e9895caccfafe92baed160e0bcad0cf4da771c61a75007d43f18: Status 404 returned error can't find the container with id 4fb02a212d15e9895caccfafe92baed160e0bcad0cf4da771c61a75007d43f18 Oct 30 19:17:19.906996 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:19.906976 4862 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:17:20.047041 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:20.047005 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj67d/must-gather-fg7h2"] Oct 30 19:17:20.051677 ip-10-0-128-246 kubenswrapper[4862]: W1030 19:17:20.051652 4862 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8d0b230_a84e_4814_86bb_89794ed809b5.slice/crio-240b2d1f32f0651924d61acf64c14c06e6d76d28661cb8c6815fe5bae0c3485e WatchSource:0}: Error finding container 240b2d1f32f0651924d61acf64c14c06e6d76d28661cb8c6815fe5bae0c3485e: Status 404 returned error can't find the container with id 240b2d1f32f0651924d61acf64c14c06e6d76d28661cb8c6815fe5bae0c3485e Oct 30 19:17:20.057318 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:20.057295 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p4hc2/must-gather-7qp95" event={"ID":"83cd0df9-68c5-4c3f-abbc-14aebea8e91c","Type":"ContainerStarted","Data":"4fb02a212d15e9895caccfafe92baed160e0bcad0cf4da771c61a75007d43f18"} Oct 30 19:17:21.088733 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:21.088700 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj67d/must-gather-fg7h2" event={"ID":"d8d0b230-a84e-4814-86bb-89794ed809b5","Type":"ContainerStarted","Data":"240b2d1f32f0651924d61acf64c14c06e6d76d28661cb8c6815fe5bae0c3485e"} Oct 30 19:17:22.104067 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:22.104024 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj67d/must-gather-fg7h2" event={"ID":"d8d0b230-a84e-4814-86bb-89794ed809b5","Type":"ContainerStarted","Data":"19f12ba1f40ed8e58fb767768b8f7337682fa170b5aff2fcac5772d24292ba1c"} Oct 30 19:17:22.104510 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:22.104087 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj67d/must-gather-fg7h2" event={"ID":"d8d0b230-a84e-4814-86bb-89794ed809b5","Type":"ContainerStarted","Data":"11431d82199905ecd6b4580b5047e8270f0c45aa40849817beaf6a680fd93eb5"} Oct 30 19:17:24.350745 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:24.350708 4862 scope.go:122] "RemoveContainer" containerID="45c7dbeb6b92e52d48a396311013ff70c89e3014c478a4c764cf0b8e915bfbb6" Oct 30 19:17:25.016885 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:25.016813 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dt49r_5a8820c7-364d-4f01-90e2-ea9979976e47/global-pull-secret-syncer/0.log" Oct 30 19:17:25.641135 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:25.641101 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6cmjc_f11682e3-9202-425f-af04-da7b7c2351be/connection-checker/0.log" Oct 30 19:17:25.786304 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:25.783549 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6dk47_50098568-1891-40d1-886a-9065582c2467/connection-checker/0.log" Oct 30 19:17:25.941523 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:25.925979 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-qnth8_5a311faf-5bcd-47ab-b102-62aa0abf168c/connection-checker/0.log" Oct 30 19:17:26.455285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:26.448542 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vz7gh_7f619d36-116d-4434-bf1e-1aecb4d7fefe/konnectivity-agent/0.log" Oct 30 19:17:26.580285 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:26.569614 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-246.ec2.internal_46fe009f9a7f834e879e5480aab152d6/haproxy/0.log" Oct 30 19:17:28.178786 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:28.178708 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p4hc2/must-gather-7qp95" event={"ID":"83cd0df9-68c5-4c3f-abbc-14aebea8e91c","Type":"ContainerStarted","Data":"07e7b5227c927c764b2ccc4360dd87ddc34f887484baf3693ee4c1a6e6320299"} Oct 30 19:17:29.198070 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:29.196771 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p4hc2/must-gather-7qp95" event={"ID":"83cd0df9-68c5-4c3f-abbc-14aebea8e91c","Type":"ContainerStarted","Data":"60b1b1d4acc8f8e586a79add872daa3044716bd62e812413e0d48f21afa5472c"} Oct 30 19:17:29.256144 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:29.254180 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-p4hc2/must-gather-7qp95" podStartSLOduration=3.087779887 podStartE2EDuration="10.25416225s" totalImagesPullingTime="7.166382363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:29.250582442 +0000 UTC m=+4748.299627357" watchObservedRunningTime="2026-10-30 19:17:29.25416225 +0000 UTC m=+4748.303207162" Oct 30 19:17:29.256870 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:29.256807 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wj67d/must-gather-fg7h2" podStartSLOduration=9.419093605 podStartE2EDuration="10.25679419s" totalImagesPullingTime="837.700585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:22.161990398 +0000 UTC m=+4741.211035314" watchObservedRunningTime="2026-10-30 19:17:29.25679419 +0000 UTC m=+4748.305839105" Oct 30 19:17:32.313894 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:32.313822 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/0.log" Oct 30 19:17:32.828182 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:32.828090 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/1.log" Oct 30 19:17:33.286433 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:33.286333 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9jr6v_e7d0e911-850b-46bc-aabe-237660e9dbaa/manager/0.log" Oct 30 19:17:33.332456 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:33.332416 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7rt9f_7afa1539-e989-4bdd-bdda-eccf8ae776c3/proxy/0.log" Oct 30 19:17:34.662985 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:34.662950 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5748484d8c-wtztm_8db3b37b-c28b-4b43-8e5d-7e285f8a6819/authorino/0.log" Oct 30 19:17:35.028615 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:35.028514 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff"] Oct 30 19:17:35.037670 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:35.037624 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f7c159-6q4ff"] Oct 30 19:17:35.076286 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:35.075741 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1700b0-10fd-4357-96b4-f665910f6ebb" path="/var/lib/kubelet/pods/bf1700b0-10fd-4357-96b4-f665910f6ebb/volumes" Oct 30 19:17:35.981294 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:35.980602 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-rlk29_f3204144-0813-45d0-a202-acb52f0aeaa1/limitador/0.log" Oct 30 19:17:36.237238 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.237149 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/0.log" Oct 30 19:17:36.670409 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.670374 4862 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx"] Oct 30 19:17:36.677377 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.676560 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.713605 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.713570 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx"] Oct 30 19:17:36.736991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.736957 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/1.log" Oct 30 19:17:36.807632 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.807590 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-sys\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.807824 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.807724 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-podres\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.807824 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.807751 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-proc\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.807917 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.807857 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj72l\" (UniqueName: \"kubernetes.io/projected/33723e25-10b6-49ab-b3f7-c91c583d44ef-kube-api-access-sj72l\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.807961 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.807927 4862 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-lib-modules\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.909431 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.909390 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-lib-modules\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.909852 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.909826 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-sys\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.910009 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.909984 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-lib-modules\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.910100 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.909737 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-sys\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.910165 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.910099 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-podres\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.910165 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.910128 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-proc\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.910295 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.910172 4862 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj72l\" (UniqueName: \"kubernetes.io/projected/33723e25-10b6-49ab-b3f7-c91c583d44ef-kube-api-access-sj72l\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.910438 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.910419 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-podres\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.910597 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.910575 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33723e25-10b6-49ab-b3f7-c91c583d44ef-proc\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:36.939860 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:36.939774 4862 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj72l\" (UniqueName: \"kubernetes.io/projected/33723e25-10b6-49ab-b3f7-c91c583d44ef-kube-api-access-sj72l\") pod \"perf-node-gather-daemonset-kmgrx\" (UID: \"33723e25-10b6-49ab-b3f7-c91c583d44ef\") " pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:37.015959 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:37.015127 4862 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:37.371998 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:37.371936 4862 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx"] Oct 30 19:17:37.403294 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:37.400356 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9jr6v_e7d0e911-850b-46bc-aabe-237660e9dbaa/manager/0.log" Oct 30 19:17:37.467321 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:37.465235 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7rt9f_7afa1539-e989-4bdd-bdda-eccf8ae776c3/proxy/0.log" Oct 30 19:17:37.550281 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:37.545758 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/0.log" Oct 30 19:17:38.001849 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:38.001170 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/1.log" Oct 30 19:17:38.328741 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:38.326824 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" event={"ID":"33723e25-10b6-49ab-b3f7-c91c583d44ef","Type":"ContainerStarted","Data":"30235e930434698fcc7bf6bc18b6288779c58eeea81dd5c917c2990c63a2c803"} Oct 30 19:17:38.328741 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:38.326966 4862 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" event={"ID":"33723e25-10b6-49ab-b3f7-c91c583d44ef","Type":"ContainerStarted","Data":"8c3025edb1a91eacfef78ce68c1ecfe76fe13ef5993594c11dbc0899ba807141"} Oct 30 19:17:38.328741 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:38.327004 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:38.365987 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:38.365924 4862 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" podStartSLOduration=2.365908998 podStartE2EDuration="2.365908998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:38.365022007 +0000 UTC m=+4757.414066931" watchObservedRunningTime="2026-10-30 19:17:38.365908998 +0000 UTC m=+4757.414953912" Oct 30 19:17:38.525070 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:38.525043 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9jr6v_e7d0e911-850b-46bc-aabe-237660e9dbaa/manager/0.log" Oct 30 19:17:38.568540 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:38.566554 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7rt9f_7afa1539-e989-4bdd-bdda-eccf8ae776c3/proxy/0.log" Oct 30 19:17:38.858572 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:38.858541 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-8g5tv_874e6413-424d-4443-80fc-f9e4d0a63b6c/cluster-monitoring-operator/0.log" Oct 30 19:17:39.611663 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:39.611626 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/0.log" Oct 30 19:17:39.997635 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:39.997548 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/1.log" Oct 30 19:17:40.050719 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:40.050688 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m"] Oct 30 19:17:40.060644 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:40.060614 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7418fe-5mk5m"] Oct 30 19:17:40.287348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:40.286757 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wmpk_f2f8a427-9fa5-43c2-b914-801d52dbc3cf/node-exporter/0.log" Oct 30 19:17:40.419150 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:40.419114 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wmpk_f2f8a427-9fa5-43c2-b914-801d52dbc3cf/kube-rbac-proxy/0.log" Oct 30 19:17:40.550958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:40.550928 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wmpk_f2f8a427-9fa5-43c2-b914-801d52dbc3cf/init-textfile/0.log" Oct 30 19:17:40.607631 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:40.607598 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9jr6v_e7d0e911-850b-46bc-aabe-237660e9dbaa/manager/0.log" Oct 30 19:17:40.613589 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:40.613564 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/0.log" Oct 30 19:17:40.661073 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:40.661043 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7rt9f_7afa1539-e989-4bdd-bdda-eccf8ae776c3/proxy/0.log" Oct 30 19:17:41.050790 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:41.050736 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/1.log" Oct 30 19:17:41.060217 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:41.060178 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62015e36-11c0-4b1e-ba85-d76c99717575" path="/var/lib/kubelet/pods/62015e36-11c0-4b1e-ba85-d76c99717575/volumes" Oct 30 19:17:41.607348 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:41.607309 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9jr6v_e7d0e911-850b-46bc-aabe-237660e9dbaa/manager/0.log" Oct 30 19:17:41.689864 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:41.689815 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7rt9f_7afa1539-e989-4bdd-bdda-eccf8ae776c3/proxy/0.log" Oct 30 19:17:43.682730 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:43.682698 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/0.log" Oct 30 19:17:44.216079 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:44.216047 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jzvfr_58092d26-be96-4cc3-8584-cb9b729475dd/manager/1.log" Oct 30 19:17:44.345438 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:44.345409 4862 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wj67d/perf-node-gather-daemonset-kmgrx" Oct 30 19:17:44.872007 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:44.871970 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9jr6v_e7d0e911-850b-46bc-aabe-237660e9dbaa/manager/0.log" Oct 30 19:17:44.914991 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:44.914955 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7rt9f_7afa1539-e989-4bdd-bdda-eccf8ae776c3/proxy/0.log" Oct 30 19:17:45.706411 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:45.706382 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc6d98d7-wrwtb_c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e/kube-auth-proxy/0.log" Oct 30 19:17:45.783493 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:45.783452 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-46d6l_6318be39-5955-4db1-99f2-318432ad0e0d/istio-proxy/0.log" Oct 30 19:17:45.855547 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:45.855514 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-mk86v_62fc3e8a-9a9e-4b43-9cc7-dee3cdcb97ce/networking-console-plugin/0.log" Oct 30 19:17:46.027281 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:46.027035 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5"] Oct 30 19:17:46.039682 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:46.037370 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1120e8d-d2bf5"] Oct 30 19:17:46.253779 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:46.253499 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ff9667845-fjrpd_5b5e199e-2896-450f-9e73-838fb7e12e48/router/0.log" Oct 30 19:17:47.065145 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:47.064966 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65b1b3b-b94c-457c-ab38-712fa4ad7d13" path="/var/lib/kubelet/pods/d65b1b3b-b94c-457c-ab38-712fa4ad7d13/volumes" Oct 30 19:17:47.341083 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:47.340962 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/2.log" Oct 30 19:17:47.348774 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:47.348690 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gxwj6_21caa3cf-ca4f-424a-9f40-6f91a7b176c3/console-operator/3.log" Oct 30 19:17:49.600298 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:49.600240 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc6d98d7-wrwtb_c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e/kube-auth-proxy/0.log" Oct 30 19:17:49.671304 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:49.670036 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-46d6l_6318be39-5955-4db1-99f2-318432ad0e0d/istio-proxy/0.log" Oct 30 19:17:49.889555 ip-10-0-128-246 kubenswrapper[4862]: E1030 19:17:49.889456 4862 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-f2zht" containerName="extractor" Oct 30 19:17:50.026969 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:50.026932 4862 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt"] Oct 30 19:17:50.036832 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:50.036799 4862 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a615556-ccrzt"] Oct 30 19:17:50.091958 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:50.091906 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ff9667845-fjrpd_5b5e199e-2896-450f-9e73-838fb7e12e48/router/0.log" Oct 30 19:17:50.252766 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:50.252685 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-lkc5k_0bfbeb5b-8284-4324-a8d0-95d95d23dfc6/volume-data-source-validator/0.log" Oct 30 19:17:51.062179 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:51.062134 4862 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb9bd28-af5e-40a7-985f-509d0bded58c" path="/var/lib/kubelet/pods/bbb9bd28-af5e-40a7-985f-509d0bded58c/volumes" Oct 30 19:17:52.092650 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:52.092619 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5748484d8c-wtztm_8db3b37b-c28b-4b43-8e5d-7e285f8a6819/authorino/0.log" Oct 30 19:17:52.852700 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:52.852672 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-rlk29_f3204144-0813-45d0-a202-acb52f0aeaa1/limitador/0.log" Oct 30 19:17:53.408136 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:53.408105 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r94b8_529241e2-eb4e-4412-ad1f-37dee4aa9b65/dns/0.log" Oct 30 19:17:53.535288 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:53.535234 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r94b8_529241e2-eb4e-4412-ad1f-37dee4aa9b65/kube-rbac-proxy/0.log" Oct 30 19:17:53.793693 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:53.793616 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rv2ks_99e4d7d6-f980-4b03-8200-d6c5d8d3dc40/dns-node-resolver/0.log" Oct 30 19:17:55.042292 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:55.041471 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5748484d8c-wtztm_8db3b37b-c28b-4b43-8e5d-7e285f8a6819/authorino/0.log" Oct 30 19:17:55.694062 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:55.694027 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6wlg8_4f34929e-0b21-4743-af11-1360d39f162b/node-ca/0.log" Oct 30 19:17:55.900872 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:55.900833 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-rlk29_f3204144-0813-45d0-a202-acb52f0aeaa1/limitador/0.log" Oct 30 19:17:55.999216 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:55.999098 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ffc6d98d7-wrwtb_c13dd46c-b1fa-4044-b9d0-6c2a1a5f1a6e/kube-auth-proxy/0.log" Oct 30 19:17:56.064713 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:56.064679 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76597d6855-46d6l_6318be39-5955-4db1-99f2-318432ad0e0d/istio-proxy/0.log" Oct 30 19:17:56.460651 ip-10-0-128-246 kubenswrapper[4862]: I1030 19:17:56.460621 4862 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ff9667845-fjrpd_5b5e199e-2896-450f-9e73-838fb7e12e48/router/0.log"